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About the evaluator 
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Executive summary 


The project 


IRIS is designed to improve primary school teachers’ use of dialogue and feedback through using 
video technology for collaborative teacher development with a view to improving academic outcomes 
for pupils. It is based around a video technology system (IRIS Connect) which enables teachers to 
record, edit, and comment on teaching and learning. 


In this pilot, the project comprised six ‘film club’ events each lasting two hours: three in which teachers 
reviewed lesson clips from other schools, two in which they reviewed each other's lessons, and one 
film club in which they reviewed their experience. The project aimed to create long-term, whole-school 
change, embedding the use of dialogue and feedback in school culture. This evaluation, however, 
focused specifically on the impact of IRIS Connect on the teachers attending film clubs, as stated in 
the original protocol agreed prior to the project starting. 


Schools were free to choose which teachers participated in the trial, but were encouraged to focus on 
Year 5 pupils. Teachers worked collaboratively in the ‘film clubs’ to review lesson clips from other 
schools and to plan, teach, record, and review their own lessons using the IRIS Connect online 
platform. The project was designed and supported by IRIS Connect and Whole Education with advice 
from academics at the Universities of Cambridge and Leeds, and additional content from Routledge. 


This pilot project evaluated (i) how teachers implemented the intervention, and (ii) the change in 
teachers’ thinking and practice. Teachers participating in the project were compared with other 
teachers in their schools who did not participate. The project lasted for seven months from January to 
July 2016. The first three clubs in each school were held between January and April and the second 
three between May and July. Twelve schools were initially recruited for the project, eleven of which 
participated. 


Key conclusions 


1. The overwhelming majority of teachers who responded to the survey believed that the 
intervention was a good use of their time and had improved their practice. 


2. During the seven-month pilot, three of the eleven participating schools chose not to engage with 
the second half of the project in which teachers videoed their own lessons. 


3. The intervention demanded a substantial proportion of teachers’ development time. A further trial 
should provide schools with sufficient time to build this into their annual plans and to embed the 
intervention in their plans for school improvement. 


4. Ten of the eleven participating schools reported that they would continue to use IRIS Connect 
after the pilot had ended. 


5. The training materials are well developed, clearly understood by teachers, and ready for trial. 


What are the findings? 


On the basis of a range of evidence from videos of lessons, interviews, and ‘before and after’ surveys, 
this evaluation has found: moderate evidence of change in school climate, strong evidence that film 
clubs promote discussion of teaching and learning, moderate evidence of change in teachers’ thinking 
and moderate evidence of changes in practice. 


All participating schools successfully implemented the first half of the intervention, which involved 
groups of teachers discussing video clips provided through the online platform. Three schools did not 
implement the second half of the intervention, which involved teachers videoing and discussing their 
own lessons. A large majority of teachers responding to an end-of-project survey were positive about 
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the value of the project to them, and all but one of the teachers acting as a ‘school champion’ reported 
that their school would be continuing to use IRIS Connect along the lines of this project. 


The intervention seems to operate in two ways: (i) by helping teachers to identify possible 
improvements in practice, and (ii) by helping teachers to become more comfortable with discussing 
their classroom work with others. The second of these processes takes longer to embed and this 
carries implications for the length of an efficacy trial. 


The intervention is well defined and could be replicated in the form used in this pilot. Some 
refinements could help teachers and schools to overcome some of the challenges they faced, such 
as: strengthening the online guidance to emphasise the value of teachers openly discussing the 
impact of the programme on their own practice; emphasising the value of discussion that focuses on 
why a particular example of teaching might be engaging pupils in a particular way that is fostering 
their learning; highlighting for teachers how the online platform can be used to identify and comment 
on moments in a lesson that exemplify an aspect of teaching deemed critical to learning; and 
encouraging discussion of these moments between schools. Many of these recommendations have 
been implemented by Whole Education in the time since the pilot. 


How was the pilot conducted? 


A process evaluation examined the implementation of the intervention and its impact on the thinking 
and practice of teachers in the pilot as compared with teachers in the same schools who did not 
participate. The evidence used in the evaluation included usage of data from the online platform, 
lesson videos, reports and videos of film clubs, focus groups, interviews, and surveys of teachers 
before and after the project, including a comparison group of teachers who did not take part in the 
pilot. 


Table 1: Summary of pilot findings 
Question lace lias} Comment 


Participating teachers believed the intervention 
improved their practice. This change in practice is 
supported by evidence collected through surveys 
and analysis of videoed lessons. No data on pupil 

outcomes was collected as part of this pilot. 


Moderate 

Is there evidence to support (with strong 
the theory of change? formative 

findings) 


Although the intervention demands teachers’ time, 
teachers felt it was worth it and the participating 
Was the approach feasible? Moderate schools planned to continue after the project. Three 
of the eleven schools did not fully implement the 
programme within the seven months of this pilot. 


The online platform and the materials used in this 
project are ready for systematic delivery across a 
larger number of schools. 


Is the approach ready to be 
evaluated in a trial? 
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Introduction 


Intervention 


Whole Education and IRIS Connect is a whole-school approach to using video technology for 
collaborative teacher development with a focus on teacher feedback and dialogic teaching in primary 
schools. 


Whole Education is a not-for-profit organisation that supports a large network of schools to improve 
through professional development. IRIS Connect, a technology provider, provides video technology 
and associated professional support for schools that buy a three-year subscription to their system. 
Together, both organizations aim to develop school improvement through collaborative professional 
development using video technology as a vehicle for whole-school development as well as a tool for 
improving teachers’ thinking and practice. 


Schools that joined this pilot received guidance outlined in a document ‘Teaching is Learning’ (IRIS 
Connect, 2014). This guidance draws substantially on Joyce and Showers (2002) and encourages 
schools to embed their use of IRIS Connect within a whole-school approach to improvement led by 
professional development. The guide advises schools on how to overcome teachers’ fears that the 
technology will be used to make negative evaluations of their work. The guidance also suggests that 
the school should identify ‘pathfinders’ who will champion the use of the technology. IRIS Connect 
promotes a culture of ‘open classrooms’ in which teachers learn from each other by observing and 
discussing practice. 


Schools already subscribing to IRIS Connect, and schools that had already declared their intention to 
buy a subscription, were invited to join this pilot. Schools that subscribe to IRIS Connect receive 
training on how to use the technology and how to make it central to a programme of school 
improvement. This training is provided through a two-hour training session at each school. IRIS 
Connect suggested that up to ten teachers, including the ‘school champion’, should attend this 
session. The school champion was responsible for leading and co-ordinating the use of the video 
technology in schools and for liaising with senior management to align this practice with school 
improvement objectives. They were also the key point of contact between the school, IRIS Connect, 
and Whole Education. IRIS Connect also provided on-demand troubleshooting support in the event of 
technical problems. Additional support was provided to the schools participating in the pilot. This 
support comprised project meetings held in London, online support through the IRIS Connect 
platform, and a telephone support line for advice and problem-solving (provided by IRIS Connect and 
Whole Education). Each school received a £1,000 budget for travel and supply cover costs. 


IRIS Connect technology 


Schools could opt for different quantities and types of video equipment. Three schools in this pilot 
opted for a single 360° camera. The remainder (eight) used two iPads mounted on flexible tripods, 
one of which could be used to follow the teacher, while the other could be used to capture the 
interaction and dialogue of a small group of children. Both options included the IRIS Connect online 
platform giving access to videos stored on the ‘cloud’. We found no difference in practice or outcomes 
that could be associated with choice of technology format. Classroom videos could not be directly 
shared or transmitted via the web. Access to any video was controlled by the teacher of the class who 
could choose others who were permitted to view (though never to download) videos stored in the 
cloud. The platform included a ‘cartoonising’ feature so that individual children were not identifiable. 
Teachers in different schools were able to share videos, resources, and ideas through a group space 
on the online platform: online guidance and materials were provided through this online group space. 


Anyone (including the teacher) viewing a video could add comments which were linked to exact 
moments in the video. The platform also had a facility enabling comments to be categorized using a 
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built-in ‘form’. Data entered into these forms could be analysed using standard descriptive tools. The 
platform also included an edit facility which enabled teachers to edit a video, selecting a short clip 
illustrating a theme in which they were interested. 


Improving collaborative teacher development 


The theory of intervention is outlined in Figure 1. The intervention assumes that schools will use video 
technology more effectively when it is embedded in the school’s strategy for improvement. The 
intervention also assumes a particular view of how schools improve. It takes it for granted that school 
improvement is fostered by collaboration and shared leadership in a culture which encourages 
teachers to be open about challenges they face and willing to discuss children’s learning. These 
assumptions are represented in the first two sections Figure 1. Schools which have not already 
established this kind of culture and approach to school improvement may use the introduction of the 
video technology as part of a strategy to develop this approach to school improvement. In these 
circumstances, effects of using the technology are likely to take longer to emerge for teachers and for 
learners. The intervention assumes that teachers’ use of the video technology (section 2 in Figure 1) 
will help to make teaching and learning visible to them in a way that is not routinely possible in the 
course of classroom teaching. Increased visibility of teaching and learning is expected to act as a 
stimulus for dialogue between teachers about the learning that is taking place and the ways in which 
teaching is helping or hindering this learning. It is presumed that this dialogue will lead to changes in 
teachers’ thinking and practice which will improve children’s learning (sections 3 and 4 in Figure 1). 
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IRIS Connect 


1 Change management 


Objectives 


Implementation 


to foster a collaborative approach to 
school improvement; 

to encourage planning _ for 
improvement which _ positions 
development at the centre; and 

to support school leadership in creating a 
culture in which lesson observation is 
regarded as_ non-threatening and 
supportive. 


school 
staff 


positioning the intervention within Whole 
Education’s programme of support for a 
network of schools; and 

through the content and tone of training 
provided and support materials (hard 
copy and online). 


2 IRIS Connect technology 


Objectives 


Implementation 


to provide reliable video technology that 
is easy to use in the classroom and easy 
to use in reviewing lessons; 

to help teachers to become more 
confident in sharing videos of their 
lessons with colleagues; and 

to focus teachers’ collaboration on 
relationships between the practice of 


robust 
iPads; 
secure IT environment where access to 
videos is controlled by the teacher of the 
lesson; 

immediate response technical support; 
and 

online platform which is easy to use to 


technical specification using 


teaching and_ students’ classroom share comments and analysis. 
learning. 
3 Classroom practice 
Objectives Implementation 
e to enable teachers to distinguish e robust technical specification using 
between different types of teacher iPads; 


feedback to pupils; and 

to help teachers to change their practice 
in dialogic teaching and feedback so they 
make more use of the kinds of feedback 
encouraged by Hattie and Timperley’ 
review of evidence. 


secure IT environment where access to 
videos is controlled by the teacher of the 
lesson; 

immediate response technical support; 
and 

online platform which is easy to use to 
share comments and analysis. 


4 Student outcomes 
Objectives Implementation 


to enable pupils to play a more formative 
role in directing their own _ learning 


by increasing the focus of teachers’ 
feedback on pupils’ self-regulation. 
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(informed by teachers’ feedback); and 
e to improve pupils’ achievement as 
indicated by standardised assessments. 


Film clubs 


Primary school teachers in pilot schools worked collaboratively, using IRIS Connect technology, to 
review lessons with a focus on improving feedback to students. Their collaboration was organised 
through ‘film-clubs’ where they reviewed a series of lessons from other schools and their own 
teaching. 


The intervention lasted six months from January to July 2016. During this period, participating schools 
were asked to organise six film clubs, three between January and April and three between May and 
July. IRIS Connect provided videos of lessons for the first three film clubs and asked teachers to 
video their own lessons for the final two. Detailed guidance for organising these film clubs was 
provided through the online IRIS Connect platform. The film clubs were designed to develop teachers’ 
thinking and practice in relation to classroom talk and higher-order feedback by ‘making learning 
visible’. 


Resources and guidance for the film clubs were provided through the IRIS Connect online platform 
(see Appendix 2A). Film clubs typically lasted about an hour and took place at the end of the teaching 
day in time reserved for meetings, preparation and CPD. This guidance referred to Figure 2 as an 
organising framework. 
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Figure 2: Guidance on feedback and teacher-child interaction provided through the online 
platform 


Reflection and Planning 
Rich tasks and 5 7 Clear understanding of 
eee learning cltambacsialins 
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Effective feedback 


To learners From learners Between learners 


Film clubs one to three focused on classroom videos provided by IRIS Connect with support from the 
University of Cambridge and MediaMerge. Julia Snell and Adam Lefstein of the University of Leeds 
advised IRIS Connect in the design of the intervention and kindly provided two of the videos used in 
the film clubs. The online guidance suggested preparatory work for teachers along with questions the 
school champion could use to guide discussion. School champions were asked to arrange for film 
clubs four to five to focus on classroom videos provided by teachers in the school. The online 
guidance provided guidance to teachers on selecting video clips for the film clubs. There was also 
guidance to school champions on how to organise the discussion of the lessons. Following the final 
film club, schools were asked to complete a self-evaluation of their experience. 


iris 


cans aloft howe 


Film clubs one to three, between January and March, focused on short video clips of lessons on three 
aspects of dialogic teaching and feedback: classroom talk and teaching (film club one), questioning 
and group talk (film club two), and feedback (film club three). School champions were encouraged to 
prompt teachers’ thinking before the film club using resources provided through the online system. 
Guidance on running and following up each film club was also provided online (see Appendix 2A for 
details). Teachers were asked to prepare for film clubs four and five (April to June) by selecting edited 
video clips of their own lessons which they believed would be useful for discussing the ways in which 
they were using dialogue and feedback in their teaching. The support provided through the online 
platform included Hattie and Timperley’s categorisation of different types of feedback. The online 
platform also provided ‘form’ facilities that teachers could use to categorise comments which were 
added to each video clip. These clips were discussed at the film clubs and the guidance provided is 
summarised in Appendix 2A. In the final film club, teachers were asked to evaluate their experience in 
the film clubs and to discuss what they thought they had learned and how their practice had been 
influenced. Guidance provided to school champions asked for these reflections to be summarised 
using the online system. 
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Background evidence 


Video technology was used in this pilot project to improve teachers’ thinking and practice. The 
evidence that the school effect on achievement is largely due to the quality of individual teachers 
(Kane, Taylor, Tyler and Wooten, 2011; Slater, Davies and Burgess, 2012) provides a broad rationale 
for approaching school improvement in this way. 


IRIS Connect provides over 900 primary schools in England with video technology that offers an 
extensive online platform enabling teachers to share and comment on videos of teaching and 
learning. The system gives individual teachers control over who has access to videos of their 
classrooms, and anyone with access to a video can add comments about specific moments in a 
video. The system also enables teachers to devise and share ‘forms’ which can be used to classify 
observations of events or dialogue in videos. 


The proportion of state primary schools in England using IRIS Connect (930, around 5%) is 
considerably smaller than the proportion of state secondary schools using the system (723, around 
20%).' Nonetheless, these figures imply ‘promise’ in an era when policy expects schools to be 
managing their own budgets and taking responsibility for teacher development. 


MirandaNet and Preston (undated) reported results from a survey of 99 teachers in 30 schools using 
IRIS Connect. Most teachers reported improvement in their teaching skills (94%), confidence (74%), 
readiness to experiment (78%), and collaboration (81%). Just under half believed there had been 
improvements in pupils’ learning and just over 10% believed there had been improvements in pupils’ 
behaviour. The report does not explain how the sample was recruited and does not provide any 
information about how these teachers used IRIS Connect. Nor does it provide any direct evidence 
regarding outcomes for pupils. 


Variation in approaches to using video technology to improve teaching and learning 


There are two broad approaches to using video technology to promote and focus teachers’ dialogue. 
One approach is one to one coaching. This may take place through real time ‘in-ear’ coaching or 
through a teacher and a coach reviewing a video after a lesson. As Quinn et al. (2015) point out, this 
increases the opportunities for teachers to review lessons together since they can do this when no 
lessons are timetabled. The ‘Best Foot Forward’ intervention based at Harvard University (Kane et al., 
2013; Quinn et al/., 2015) aimed to use this approach to video technology to ‘de-privatise’ teaching—to 
encourage more dialogue between teachers about how they are teaching and how this is connected 
with children’s learning. A second approach to using video technology aims to improve teachers’ 
dialogues through establishing collaborative groups that discuss videos of teaching from beyond the 
school as well as videos of their own lessons. This approach is referred to as ‘video clubs’ or ‘film 
clubs’. The intervention in this pilot used the second approach. 


One of the clear benefits of using video technology to support collaborative teacher development is 
increasing and deepening teachers’ dialogue about teaching and learning (for example Borko et al., 
2008). This dialogue may be face-to-face or conducted through annotations enabled by an online 
system (Rich and Hannafin, 2009). Reports of interventions using video technology (for example 
Kane et al., 2013) have emphasised that it takes time to develop a culture in which teachers are 
willing to talk freely about their work. At first, video technology tends to be taken up by a few 
enthusiasts and teachers prefer to avoid sharing videos of their lessons with others. The tendency for 
video technology to be gradually adopted forms the basis of the IRIS Connect approach to supporting 
schools, an approach that draws heavily on the arguments of Joyce and Showers (2002) for school 
leaders to develop a culture of shared practice and experimentation. Reviews of schools’ adoption of 
video technology (see Brouwer, 2011) paint a similar picture to that observed during the development 
stage preceding this pilot. Therefore, schools’ experience with the IRIS Connect video technology is 
likely to frame the way that teachers responded to the intervention. 


' Based on figures provided by IRIS Connect and data from Department for Education (2016a). 
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Reviews of evidence of the power of video technology for teacher development are broadly positive 
about impact on teachers’ practice 


Reviews (such as Brouwer, 2011; Gaudin and Chaliés, 2015) have identified emerging evidence that 
increased skills in lesson analysis, and a deeper understanding of how teachers and learners interact 
with subject matter, has an impact on teacher effectiveness. Brouwer’s review also suggested that 
improvements were associated with a stronger understanding of the relationships between the 
practice of teaching and students’ learning and with change in teachers’ practice. 


Studies reviewed by Brouwer (Sherin and Han, 2004; van Es and Sherin, 2005; Krammer et al., 2006) 
point to these changes being a gradual process through ‘video clubs’ in which teachers learn through 
discussing videos over a sustained period, with interventions lasting up to a year. These studies have 
primarily used intensive qualitative observation to try to identify features of teachers’ collaboration 
which are critical to improvement in their teaching. For example, van Es (2012) suggested that 
teachers learned more when they focused on ‘moments of interesting thinking’. From their review of 
evidence, Gaudin and Chaliés (2015) infer that short clips of videos—which focus on a particular 
aspect of teaching and learning—are more effective than open-ended reviews of complete lessons in 
changing teachers’ thinking and practice. Their review suggests that teachers learned more when 
they had opportunity to view a clip several times, and also that teachers are likely to learn more when 
discussion is guided by an experienced facilitator who has a deep understanding of the aspect of 
teaching and learning under review and well-developed skills in guiding discussions. 


Interventions to improve feedback and dialogic learning align with evidence reviews disseminated by 
the Education Endowment Foundation 


The intervention was designed in the light of the evidence (Gaudin and Chaliés, 2015) that a focus on 
a particular aspect of teaching and learning makes it more likely that video clips will provide a 
powerful focus for professional development. Gaudin and Chaliés do not cite evidence of impact on 
pupils. Therefore, the pilot was designed to focus on an area of teachers’ practice for which evidence 
of impact on pupils is available. The EEF’s Teaching and Learning Toolkit suggests that interventions 
focused on teacher feedback have an impact equivalent to eight months’ progress, although the 
advice cautions that more recent meta-analyses indicate a more modest impact of about three 
months' additional progress or nearer four months when the approach is supported with professional 
development.? Teachers’ feedback is framed by their expectations of children’s role in learning, 
teachers’ approach to classroom dialogue, and the design of tasks they set for children. 


Film clubs provide a useful and feasible medium for teachers’ collaboration in using video technology 


Video recordings have been widely used in the context of ‘plan-teach-review’ cycles in which teachers 
meet regularly over an extended period to learn by analysing their own teaching. This pattern is 
exemplified in the literatures on ‘Lesson Study’ (Lewis, Perry and Murata, 2006) and ‘Learning Study’ 
(for example Holmavist, 2011). Lesson Study and Learning Study involve small groups of teachers in 
cycles of collaboration in which they (1) plan, teach, and review a lesson, and (2) revise the lesson 
plan in the light of experience, teach the lesson again, and review the revised lesson. A typical cycle 
would involve three lessons. In Lesson Study the teachers decide the teaching strategies they will 
use; in Learning Study teachers explicitly aim to use ‘Variation Theory’ to direct their teaching. The 
key idea in Variation Theory is that teachers must highlight what they want pupils to learn through 
multiple varying examples while keeping constant all factors irrelevant to the learning. 


The terms ‘film club’ or ‘video club’ have been coined to denote examples of ‘plan-teach-review’ which 
(1) have not explicitly referenced either Lesson Study or Learning Study and (2) have relied on 
direction from a researcher or facilitator who has taken a lead in selecting video clips for review and 
guiding teachers’ discussion (for example, Sherin and Han, 2004; van Es, 2012). The film club model 
has been used in interventions explicitly focused on dialogue and feedback in the classroom (see, for 


*https://educationendowmentfoundation.org.uk/evidence/teaching-learning-toolkit/feedback/ 
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example, Van den Bergh, Ros, and Beijaard, 2014; Grdschner, Seidel, Kiemer and Pehmer, 2015; 
Schindler, Gréschner and Seidel, 2015). These studies have reported positive impacts on teachers’ 
thinking and practice about feedback—albeit with very small samples and without claims to 
generalisation. Kiemer, Groéschner, Pehmer and Seidel (2015) also reported positive effects on 
students’ sense of autonomy and motivation to learn. However, impacts on pupil’s achievements have 
not yet been reported. This pilot intervention used school champions in place of an external expert. 
The first three film clubs also used clips from teaching elsewhere rather than the teachers’ own 
lessons. 


Research questions 


The key research questions are cross-referenced to Figure 1 in the list below. The pilot phase was 
used to judge (1) the likelihood of meaningful, sustainable, measurable change, and (2) the feasibility 
and acceptability of the intervention to schools and teachers. 


e Key research questions regarding the likelihood of the intervention having meaningful, 
sustainable, measurable change: 


1. To what extent have teachers changed their practice (asking more open questions, allowing 
pupils more time to respond, and focusing feedback on the content and process of learning) 
while making use of the IRIS Connect technology? (3) 

2. To what extent have teachers changed their beliefs about teaching and learning (regarding 
the benefits of challenges for pupils, the kind of feedback that is more likely to prompt 
learning, and the role of dialogue in pupils’ learning) while using the IRIS Connect 
technology? (3) 

3. To what extent have pupils changed their activity and role (for example, ‘exploring’ rather 
than ‘receiving’, talking as well as listening, in relation to peers as well as in relation to the 
teacher) while IRIS Connect technology has been used by their teachers? (3/4) 

4. What would be the appropriate student achievement measures to use? (4) (We were advised 
by the EEF before the pilot stage that it would not be appropriate to use the pilot to collect 
data on pupil achievement that would inform selection of achievement measures.) 


e Key research questions regarding the feasibility and acceptability of the intervention to schools 
and teachers: 


1. To what extent have schools engaged with using the IRIS Connect technology? (1, 2) 
2. What (a) time, (6) money and (c) resource costs have been incurred by schools— 
i. during the set-up and adoption of the IRIS Connect technology? 
ii. during continued use of the IRIS Connect technology following its initial 
establishment? (1, 2) 
3. To what extent do participating schools believe that their engagement with the IRIS Connect 
technology has given them value for money? (1, 2) 


Ethical review 


This pilot evaluation received ethical approval (ERN_15-0987A) from the Humanities, Arts and Social 
Science ethics committee of the University of Birmingham. In this judgement the ethics committee 
approved amendments to the research process that were proposed in response to evidence provided 
through the Development Phase (for which approval was granted in judgement ERN_ 15-0987). 


Information and consent forms for schools, teachers and parents are included in Appendix 1. 


Education Endowment Foundation 13 


Project team 


e Project Delivery 
Douglas Archibald, Whole Education 
Rosie Palmer, Whole Education 
Andy Newell, IRIS Connect 
Christophe Mullings, IRIS Connect 


e Evaluation Team 
Peter Davies (Project Lead), School of Education, University of Birmingham 
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IRIS Connect 
Methods 


Recruitment 


The target was to recruit 12 primary schools to participate in the pilot—six with at least one year’s 
experience with IRIS Connect and six that were new or relatively new to the technology. This 
sampling frame was designed to inform decisions about the time span of any future evaluation aiming 
to compare schools with and without video technology. The pilot followed a development phase in 
which 12 primary and secondary schools had participated. Each primary school in the development 
phase was invited to participate and informed that schools would be included on a ‘first come, first 
served’ basis. Six schools of the 12 agreed to take part in the pilot. Six schools that had purchased an 
IRIS Connect licence after the end of previous school year, or that were in the process of agreeing a 
contract, were invited to join the pilot. One of these schools dropped out in the first month of the pilot. 
Details of the total number of teachers in each participating school, and the number of teachers in 
each school participating in the intervention, are provided in table A5F.1 in the Appendices. 


Six schools relatively new to IRIS Connect were also recruited, two of these starting with the system 
in the first month of the pilot. Three other schools that had recently purchased an IRIS Connect 
licence were also approached, but declined to take part. The sampling aimed to recruit one-form as 
well as two-form entry schools to check the feasibility of the intervention in small as well as medium 
sized schools. The flow of new schools being recruited shortly before the start of the pilot curtailed the 
scope for including more schools with high proportions of children eligible for free school meals. 


The dates when each school became a user of IRIS Connect technology is shown in Table 2. This 
table also provides background information about the schools and the teachers participating in this 
intervention. School 12 dropped out early in the pilot phase when the school project lead left the 
school. The junior School 7 received notification of an OfSTED inspection shortly after the start of the 
pilot. They were subsequently graded ‘Inadequate’ but chose to remain in the pilot as part of their 
response to inspectors’ recommendations, though the focus of the project work was switched to their 
linked infant school. 


Schools were given freedom regarding the number of teachers and the process which led to these 
teachers participating in the project. Three schools (4, 8 and 10) required all teachers to participate. In 
four schools, teachers were asked to volunteer. In four schools the headteacher and school champion 
identified participants (see Appendix 5D for details). 


Schools were encouraged to concentrate on Year 5 classes where possible, on the basis that a full 
trial would be likely to use assessment measures for children in Years 5 or 6 and schools would be 
more willing for teachers of Year 5 children to be involved in this pilot. 


Schools subscribing to IRIS Connect commit to a code of ethical conduct which includes informing 
parents. In addition, schools participating in this pilot were asked to sign and return a memorandum of 
understanding (Appendix 1A). Schools were also asked to send a letter and opt-out form to parents 
(Appendix 1B). Schools avoided using classes where parents had opted out (for film clubs four and 
five). These ethical procedures were followed to allow teachers to share videos of lessons with 
members of the evaluation team. Film clubs four and five required teachers to create videos of their 
own lessons. The intervention encouraged teachers to share lesson videos with their colleagues 
beyond the specific expectations for film clubs four and five. 
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Table 2: Characteristics of participating schools (including date of first use of IRIS Connect) 
Date of School Age |School School % % at oo) | TAs No. of 


first use of in relaye (= Ke) |b gov.* level4 teachers ® teachers 
IRIS devpt. in KS2’ inschool oye Vat ea] oys\tlaye) 


Connect _phase?? in the 
video intervention 


technology 
1 (Sep 2011 Yes 7-11 389 Com 5 4 85 20 8 4 
2 |Feb2014 Yes 5-11 610 Com 39 95 71 33 35 |4 
3 |Apr2014 (Yes 3-11 456 Com 31 43 88 25 16 7 
4 (Jun2014 (Yes 2-11 439 Com 47 6 79 24 34 (19 
5 |Jan2015 | Yes 4-11 303 F 8 9 77 13 15 4 
6 | Jul 2015 No 4-11 503 Com 20 12 82 21 34 15 
7 | Jul 2015 No 7-11 355 Com 17 2 82 18 29 9 
8 (Sep 2015 (No 4-11 391 Com 26 5 75 21 13 (17 
9 |Oct.2015 (No 3-11 212 Ac 39 5 86 13 18 3 
10 |Jan2016 (No 5-11 422 Ac 6 1 88 18 17 |16 
11 |Jan2016 No 4-11 241 Com 18 2 59 12 15 4 
12 \nla No 3-7 252 Com 15 3 15 13 16 (in/a 

All schools 80 

in England 


‘Sch = school reference number; *Devpt’: school participated in development phase; *number of 
children enrolled in the school in 2015; *School governance (Ac = academy, Com = maintained 
community, f = Maintained Foundation); °Percentage of children eligible for free school meals in last 
six years; "Percentage of children with English not first language; ’Percentage of children achieving at 
least level 4 in reading, writing and maths at age 11; °TAs = Number of teaching assistants in the 
school. 
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Data collection 


The logic model presented in Figure 1 was developed through a series of meetings between the 
project delivery team and the evaluation team before and during the development phase which 
preceded this pilot evaluation. 


Apart from the metrics of use of the online platform, all the data was collected and analysed by the 
evaluation team. Data was not anonymised at point of collection in order to facilitate matching 
evidence from different sources. However, no individual data was shared with the delivery team. 


IRIS Connect metrics data (see Appendix 5A) 


The IRIS Connect online platform automatically records usage data. Teachers must log in to view or 
add content. Every time a teacher visits a page, views a video, uploads a video, or uses one of the 
forms provided, this is logged. Two forms of data were collected from the IRIS Connect server for the 
purposes of this evaluation: 


1. The first set of data was associated with the feedback group itself. This was used only by project 
participants (or at least those that accepted the invitation to join the EEF project group). This data 
shows when pages in the feedback group content and associated video clips have been viewed. It is 
not possible to tell whether a clip has been viewed by one individual or by a group. 


2. The second set of data relates to each individual teacher’s use of the IRIS Connect system. This 
data reveals when, and by whom, videos (and clips of videos) were created. This data does not show 
whether a teacher has shared a video with anyone other than the evaluation team, neither does it 
distinguish between lesson videos specifically prompted by the project and videos which would have 
been created anyway. 


For example, if a teacher logged on to the system in preparation for a film club and viewed several 
pages and watched a couple of video clips, this would all be recorded as part of the first data set. If 
the user subsequently attended a film club and the group accessed the system using the project 
leader’s account, any pages visited would be attributed solely to the project leader. The system would 
not have recorded how many other teachers had attended the film club. Finally, if following a film club 
a teacher recorded three videos, the second data set would record that this teacher had created three 
videos (it would not distinguish between recordings prompted by the project and recordings that would 
have been made anyway). 


These data sets provide evidence of school and teacher’ engagement with IRIS Connect (RQ 5). The 
videos of film clubs and comments added to videos also provide evidence of ways in which teachers’ 
thinking was stimulated (RQ 2), and videos of successive lessons taught by the same teacher offer 
insights into possible changes in practice (RQ 1). Although this data was collected by the project 
delivery team, it was collected automatically by the online platform to provide a record of use. 


Project leads’ summaries of film clubs (see Appendix 4A) 


Schools were given an option of providing evidence of film clubs through a summary written by the 
project lead (rather than through sharing a video of the film club). This option was provided for 
schools that were in the early stage of using the IRIS Connect technology where some participating 
teachers were feeling cautious about sharing their work online. This evidence provided much less rich 
data than the videos of film clubs, but it helped to fill gaps that would have been created had we 
obtained no information about the implementation of film clubs. 
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Project lead focus group (see Appendix 3D) 


During the meeting for project leads held mid-way through the project (22 April) we conducted a short 
focus group. Eight school project leads attended this meeting (schools 2, 3, 4, 6, 7, 8, 9 and 10). One 
additional teacher attended from each of schools 2 and 3. The focus group was used to gather data 
on (1) variation in the way that schools were implementing the project and (2) challenges and costs 
that were being faced (RQ 5-7). The focus group was recorded and transcribed. 


School champion survey (see Appendix 3C) 


School champions were surveyed at the end of the pilot to gather data on schools’ engagement with 
the intervention, costs incurred, challenges that had been faced, and measures that had been taken 
to address those challenges (RQ 5-7). 


Telephone interviews with teachers (see Appendix 3E) 


We conducted short—roughly ten-minute—telephone interviews with eight teachers (who were not 
project leaders) during June and July to check our interpretation of the school’s engagement with the 
project and video clips that had been shared with the evaluation team. Interviews were recorded and 
transcribed. 


Surveys of all teachers in each school: January and July (to help answer RQs 1, 2, 3 and 5—see 
Appendices 3A, 3B) 


All teachers in each pilot school were asked to complete a questionnaire administered in January and 
again in July (see Appendices 2A and 2B). Schools were asked to include teachers who were not 
participating in the intervention to provide indicators of change that might be attributable to the 
intervention. 


Questions 1-8, 8a, 8b, 9a and 9b of the baseline survey were designed to gather evidence of 
teachers’ thinking and practice regarding feedback. These questions were informed by Hattie and 
Timperley’s (2007) review and by the development phase that preceded the pilot. Question 10 (first 
four items) was designed to gather evidence about the school context for using IRIS Connect 
technology in the light of the expectation that schools would typically take time to adjust their culture 
to facilitate effective collaboration in sharing and reviewing lessons. The final three parts of question 
10 and questions 11 and 12 were designed to gather evidence of teachers’ engagement with IRIS 
Connect and the feasibility of the intervention. 


Shared teacher videos and discussion (see Appendix 5E) 


The IRIS Connect online platform enabled teachers to share videos with the evaluation team. These 
shared video clips included any comments that teachers had added and any analysis they had 
conducted using the ‘forms’ facility on the online system. Teachers shared videos with the evaluation 
team with no mediation by IRIS Connect. 


This data was gathered to help evaluate the extent to which teachers had engaged with IRIS 
technology (RQ 5) and to focus on feedback and dialogic learning (RQs 1 and 2). The video clips and 
comments also provided insights into any changes in teachers’ thinking and practice, and ways in 
which such changes might be related to their use of the IRIS Connect technology. 
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Timeline 


Table 3: Timeline 


Date Activity 


Oct-Dec 2015 Recruitment of schools for the pilot managed by Whole Education 


One day meeting for project leads explaining the film clubs, online support, 
Jan 2016 ; ; 
and reporting requirements 


Jan 2016 Baseline survey of teachers in pilot schools (Appendix 2A) 
Jan-Mar 2016 Schools implement film clubs 1—3 
22 Apr 2016 Focus group with project leads (no members of the delivery team present) 


Jan—Jul 2016 Metrics of teachers’ use of the IRIS Connect online platform gathered 
automatically 


Mar—Jul 2016 Teacher videos shared with the evaluation team through the IRIS Connect 
online platform 
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Findings 


Evidence to support theory of change 


This intervention intended to improve children’s achievement by changing teachers’ thinking and 
practice. In this respect, the intervention is a long-term investment in teachers and schools whose fruit 
is expected to come over a period of years rather than as a direct consequence of one particular 
initiative. It assumes that schools need to provide a climate of ‘open dialogue about teaching and 
learning’ which will foster change in teachers’ thinking and practice, and works towards this end. In 
this respect, the effect of the intervention is expected to be ongoing. 


Consequently, the theory of change which justifies this intervention has many steps (as outlined in 
Figure 1). This section reviews the evidence this pilot provided about steps one to three in Figure 1. 
Videos of classrooms which were shared with the evaluation team provide some indicators of change 
in children’s practice, but the focus of the evaluation was upon change in teachers. 


IRIS Connect and the development of a climate of open dialogue (‘embedding’) (1 in Figure 1) 


Participants in the project were more likely than other teachers in their school—according to a 
baseline survey—to ‘learn a lot from their colleagues’. This probably reflects the voluntary nature of 
participation in four of the schools, and the basis for teacher selection in five others. It should be 
noted, however, that there was no easily discernible difference between the impact on teachers in 
project schools where all teachers participated compared to schools where a small (self-) selected 
group participated (see Appendix 5). 


There was no detectable change in the likelihood that participants would ‘learn from their colleagues’, 
however participants did report feeling less pressure when colleagues discussed their teaching (see 
Table A5F.3), and non-participants expressed the same view. This could indicate a spillover effect of 
the project or it might reflect a change in the school associated with willingness to take part in the 
project. At the end of the project, roughly one third of participants and 40% of non-participants 
asserted that they felt under pressure when colleagues discussed their work (Table A5F.3). 


Survey responses were higher than is typical: 80% of all teachers responded to at least one of the 
baseline and endpoint surveys, and 43% of teachers responded to both (Table A5F.1). These high 
return rates may be biased by self-selection, but nonetheless the majority of teachers—in the end-of- 
project survey—indicated that they would continue to use IRIS Connect to review their teaching. This 
suggests that the project had some success in generating a sustainable change in the opportunities 
for teachers to share and discuss their classroom experience. 


The intervention requires schools to create time for teachers to meet together—either within or 
beyond designated preparation and development time. Since most schools plan programmes of 
continuous professional development a long time in advance, the pilot did create some challenges for 
schools that had not allocated time for this kind of activity (see comments from school champions in a 
focus group at the mid-point in the project, Appendix 5B). School champions believed that the scope 
for the intervention was contingent on whether school senior leaders were committed to the idea of 
school improvement through teacher development through open dialogue. 


School champions also believed that schools needed time to help teachers to overcome fears that the 
technology would be used to their disadvantage. They were reluctant to put a firm timescale on this 
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but implied it could well take a substantial part of a school year. Nonetheless, the metrics data 
collected by IRIS Connect revealed no difference in the number of online platform hits per teacher 
between experienced schools and schools new to IRIS Connect (see Table 4). 


The final two rows of Table 4 compare the use of the online platform in schools adopting a whole- 
school approach and schools in which participation was restricted to a small group of teachers. A 
majority of schools (including those with prior experience of IRIS Connect) opted for the small group 
approach (see Appendix 5D). On average, each participating teacher in these schools used the online 
platform more than teachers in schools adopting a whole-school approach. In the latter case, 
however, the average teacher used the online platform nearly three times as much as the average 
teacher in the ‘small group’ schools (see Table 4). 


Table 4: Number of separate occasions on which the platform was accessed (per participant or 
teacher) over the whole pilot project period 


Number of Number of 


School School online online 
School new to adopted platform platform 
IRIS whole-school _ hits per hits per 
Connec participation project teacher in 
t (=1) (x5) participant dalemeaxeareye)| 
1 0 0 121 28 
2 0 0 64 7 
3 0 0 23 10 
4 0 1 81 81 
5 0 0 45 19 
6 1 0 199 47 
7 1 0 34 20 
8 1 1 47 47 
9 1 0 27 11 
10 1 1 37 37 
11 1 0 44 15 
Schools new to IRIS Connect 65 29 
Schools with pre-project 67 29 
experience 
Schools with whole-school 55 55 
approach 
Schools with small group 70 20 
approach 


Teachers in the three schools (2, 5 and 9) with the lowest rate of ‘hits on the online platform’ also 
created no videos in the second half of the intervention. This might be interpreted as showing that the 
climate in these schools had not yet become sufficiently supportive of using the system to review 
teachers’ lessons. However, two of these schools had been using IRIS Connect for nearly a year 
already, albeit with a very low rate of use (see Table A5A.1). There is an indication that the pattern of 
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use is more strongly influenced by the existing school culture and organisation than by a process of 
gradual acclimatisation as teachers get used to the system (as reported by school champions). 
School champions believed that willingness to discuss practice was helped by the sequence of film 
clubs focusing on others’ lessons followed by film clubs in which they shared their own lessons. They 
also believed that the focus on feedback and classroom dialogue was consistent with schools’ 
improvement plans (Appendix 5C). Review of videos and school champion reports of film clubs 
showed substantial variation in the degree to which film clubs strictly adhered to the guidance 
(Appendix 2). There was also variation in the openness of discussion, and the application to teachers’ 
practice. There is an indication here of a two-way relationship between schools’ general climate for 
teacher collaboration and the conduct of film clubs (see Appendix 5D). 


Film clubs as a way of promoting review of teaching and learning (2 in Figure 1) 


The spikes in the metrics data (use of the online platform, see Figures A5A.1—11) reflect the level of 
platform use around the time of film clubs. The end-of-project survey (Table A5F.4) also shows that 
teachers did engage in viewing and discussing the videos from other schools and colleagues in their 
school. Just over 70% of participants reported in the end-of-project survey that film clubs had made 
them ‘more likely to reflect on my own practice’. The majority of school champions reported that 
teachers positively engaged with film clubs (Appendix 5G) and teachers interviewed at the end of the 
project reported that the videos had stimulated discussion. There is strong evidence from a range of 
sources (the spikes in metrics data, the end-of-project survey, and the reports from school 
champions) that film clubs stimulated discussion of teaching and learning. School champions reported 
that the level of online support for these discussions was about right (see Appendix 5C). 


The shared videos of film clubs and school champions’ reports of videos showed that the discussion 
in these meetings did focus on feedback and dialogue in the way intended by the online guidance 
(see Appendix 5D). Teachers identified difference in practice and the merits of different practices. For 
example, teachers commented on the use of ‘wait time’ after a teacher had asked a question, and 
they also commented on what they saw as a somewhat surprisingly low level of praise—praise 
concentrating on successful task performance rather than personal character traits. Film club 
discussions paid less attention to the rationale for different practices. Relationships between teaching 
and learning tended to be discussed in general terms rather than through examination of relationships 
between specific teacher actions and specific examples of learning which might be inferred from the 
videos. 


In some schools, the film club discussions of videos from other schools (in the first term) kept very 
close to the guidance provided through the online platform (Appendix 2) and, in these schools, 
teachers made relatively few comparisons with their own practice. More open discussion was 
encouraged in other schools and more time was spent in these discussions considering implications 
for the teachers’ own practice (Appendix 5D). 


School champions believed that discussions were more effective when teachers had devoted more 
time to prepare for the film club. However, they acknowledged a tension between the gains from more 
preparation and the pressure on teachers’ time (see Appendices 5B and 5C). 


Teachers’ dialogue regarding teaching and learning, and change in their thinking (2C in Figure 
1) 


The comparison between baseline and end-of-project surveys provides evidence of change in 
participating teachers’ thinking about feedback. The self-characterisation feedback categories (Table 
5) were designed for the evaluation on the basis of evidence from Hattie and Timperley (2007). The 
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right and left hand statements in Table 5 were designed to avoid signalling any obvious ‘right’ answer. 
The baseline survey (before the start of the project) indicated that participants were more likely than 
non-participants to assert that their feedback practice was directive—that they were more likely to 
assert that their feedback typically ‘made plain the sequence of steps a child should follow in solving a 
problem’, ‘tells the child how close their answer was to my answer’, and ‘tells a child what the problem 
is with their thinking’. By the end of the project, participants were more likely (than at the start) to 
assert that their feedback ‘presented children with serious challenges’, ‘highlighted differences 
between alternatives’, ‘reflected their judgement about children’s reasoning’, and ‘helped children to 
see different ways of seeing a problem’. A comparison of change in participants’ thinking with change 
in non-participants’ thinking suggested four differences with modest effect sizes operating in the 
direction encouraged by the intervention (Table 5, which is also included in the Appendices as A5F.2). 


Table 5: Teacher self-characterisation of feedback (reduced scale) 


Effect size T 
((larelalel-m cele 
Left Neutral Right participants 
relative to non- 
participants) 
Baseline Participant 417 150 433 0 i 0.12 
1. My feedback Non 4 neg 
typically provides the Participant 51.6 194 | 29.0 ypically 


presents the 


child with a simple ._, |Participant i i 
next step they can | Endpoint 220 420 | Oe sliver . 
easily achieve. Non 
ee 50.0 15.6 | 34.4 challenge. 
Participant 
Baseline Participant 40.0 20.0 | 40.0 -0.03* 
2. My feedback Nlon Feedback 


carefully highlights 
the differences ae 
between alternative Endpoint Participant = 549 206 = 25.4 
ways of solving a 


35.5 | 226 | 41.9 | tolcally gives 


pupils a correct 
way of solving a 


Participant 


Non problem. 
problem. Participant 56.3. 9.4 34.4 
Baseline | Participant 25.0 11.7 63.3 My feedback 0.42 
3. My feedback Non typically 
typically makes plain Participant 9.7 32.3 58.1 concentrates on 
the sequence of helping a child 


steps a child should Endpoint Patticipant = 586 270 44.4 


follow in solving a 


to review the 
steps they have 


Non 
problem. Participant 32.3 | 35.5 32.3. taken tosolvea 
problem. 
Baseline Participant 57.6 | 15.3 | 27.1 0.29 
Non 
Participant 61.3 16.1 22.6 
4. My feedback t bh learn 
typically leaves the Participant he ae dtc 
child feeling good | Endpoint 492) 206: | 802 (|< ane whether 
about what they i: A 
have done. ey ave come 
i a good job. 
on 
Participant 74.2 9.7 16.1 
5.My feedback | Baseline | Participant 39.0 39.0 | 22.0 My feedback -0.48t 
generally reflects my Non typically tells 
judgement about Participant 65.5 20.7 13.8 the child how 
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why a child has ._, |Participant close their 
en a particular Endpeins pie exe te answer was to 
answer. Non 53.1 28.1 18.8 my answer. 
Participant 
Baseline | Participant 20.0 3.3 76.7 Kiy-feedback 0.36 
6. My feedback Non : 9.7 3.2 87.1 _ typically helps a 
typically tells a child Participant child to work out 
what the problemis | Endpoint Participant 11.1) 143 | 746 what the 
with their thinking. problem is with 
a 15.6 12.5 | 71.9 _ their thinking. 
Participant 
Baseline Participant 50.0 10.0 | 40.0 0.12% 
Non 
Participant 51.6) 16.1 32.3 
7. My feedback My feedback 
typically concentrates on 
concentrates on Participant making sure 
helping children to | Endpoint 57.1 23.8 19.0 that children 
understand different know what they 
ways of seeing a are expected to 
problem. Non do. 
Participant 56.3 21.9 21.9 


t calculated by Cohen’s d 
+ The intervention aimed to increase use of feedback described by the left hand side of this row, so for this row a 
negative effect size is desirable. 


This is encouraging for the intervention, although in two cases it looks like the effect is driven by 
changes in the thinking of non-participants away from the desired feedback characteristics. Without 
further evidence that could help us to understand this change we are not able to speculate on 
possible causes. 


Teachers’ thinking and practice (3 in Figure 1) 


In the end-of-project survey, over 95% of participants (80 out of 84) asserted that they had changed 
their practice as a consequence of the intervention (Appendix 5E open responses). Change in 
approach to using questions was the most frequently cited area of practice but they also referred to 
changes in feedback. Participants interviewed at the end of the project (Appendix 5H) expressed an 
equally positive view of the effect of the intervention on their practice, referring chiefly to feedback, 
dialogue, and pupil ownership of classroom talk. School champions asserted in their feedback to the 
project team at the mid-point in the project that changes in teachers’ practice were already in 
evidence (Appendix 5C), however they implied that they were expecting more substantive changes by 
the end of the intervention. Participants believed their practice had changed: they cited more 
collaboration, more open questions and dialogue with children, and more emphasis on feedback 
which shared responsibility with the pupil (see Appendices 5F and 5H). 


Evidence of teachers’ practice was gathered through 43 lesson videos that were shared by teachers 
with the evaluation team through the online platform (see Appendix 5E). This data needs to be treated 
with some caution as we are not able to compare teaching practices with those in place before the 
pilot. Moreover, since teachers chose whether to share lesson videos, it is likely that their selection 
was somewhat affected by ‘social acceptability bias’ as they became familiar with the intentions of the 
intervention. Finally, the videos were shared during the course of the project and, therefore, cannot be 
treated as evidence of teachers’ practice at the end of the project. Nonetheless, between 50% and 
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75% of the lesson videos were rated as displaying an approach to feedback in line with aims of the 
intervention (positive scores for items 1, 3, 4 and 6 in Table 6, and negative figures for items 2, 5 and 
7). The most positive ratings were for ‘poses serious challenges’ and ‘leaves the judgement to the 
pupil’. This means that the self-reported change in teachers’ practice (Appendix 5F) aligned with our 
analysis of lessons. Given the evidence suggesting that the type of feedback matters for pupils’ 
learning, the project evidence is encouraging for an intervention that aims to improve pupils’ learning. 
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Table 6: Characterisation of teachers’ feedback visible in shared lesson videos using the same 
format as the teacher survey (also provided in Appendix 5 as Table A5E.3 with additional 
commentary) 


Strength of tendency Slice yale) amelmsiare(calen] 
towards left hand iCo)’\ c= 1xe(smale|almar= tale! 
statement statement 
1 Gives easy/simple 2 Poses serious 
next steps challenges 
2 Highlights differences : 
between alternative 5 cael d 
responses/method i 
3 Gives correct series 5 Reviews pupil’s 
of steps or points steps or points 
Leaves the 
4 Makes pupil feel good 2 judgement to the 
pupil 
5 Reflects the teacher's Tells the pupil how 
judgement about pupil’s 9 close they were to 
current understanding teacher’s answer 
: Helps the pupil to 
6 Tells a pupil the 
problem with their 4 ou eS oa 
method/thinking ew on 
thinking/method 
7 Helps pupils . 
understand different 1 Bite ra sel 
pe aseh pede ? 0 expected to do/say 


Lesson videos shared by teachers at schools where film clubs were rated as ‘open and applied’ 
(where teachers debated the extent to which their own lessons exhibited similar characteristics and 
also whether they should change their practice) tended to display characteristics of feedback more in 


line with the aims of the intervention (see Appendix 5E). 


Feasibility 


The feasibility of the intervention can be judged by considering the following questions: 


1. Did the schools make use of the technology to the extent that was envisaged? 


2. How did school champions rate the feasibility of the project? 


3. Do the schools and participants intend to continue using the technology in a similar way after 


the project? 


4. How much time does it take? 


5. What did participants say about the value to them of using time in this way? 


6. What did participants say about improving the practicability of the project? 
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We will now consider these six questions. 


1. Teachers’ use of the online platform (Figures A5A.1—11) was very high. Over the six months 
of the project, the average rate of hits on the online system per participant varied between schools 
from 23 to 199 (overall average 66). The average rate of hits per teacher in each school (including 
non-participants as well as participants) was between 7 and 81 (overall average 30). The attrition rate 
was low. One school (12) dropped out very early in the pilot when the school champion left the 
school. School 11 (Figure A5A.11) was inspected by OfSTED during the early weeks of the 
intervention and considered dropping out of the pilot, but after receiving a disappointing grade from 
OfSTED the school chose to use its participation in the project as a means of addressing priorities for 
development and this is reflected in the high use figures in the summer term. Ten of the schools 
carried out film clubs one to three, and the remaining school conducted two of these three film clubs. 
We are not aware of any individual teacher dropping out of the project within their school although 
there were, of course, some absences from particular events. However, use of the online system 
tailed off during the summer term in four of the schools (3, 5, 8 and 9) during the period when 
teachers were expected to record and share their own lessons. A bigger concern lay with the number 
of videos being created during the summer term. Teachers in three schools (Figure 3, see schools 2, 
5 and 9) created no videos during this period. We were advised by the headteacher in school 5 that 
this was a consequence of staff turnover. Otherwise, there was no discernible pattern in the type of 
school which made less use of the technology during the second half of the project. 


Figure 3: Number of videos created per teacher over three months before and during the 
intervention* 
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t The ‘before’ figure is based on a three-monthly average for the period in which the school had IRIS Connect 
equipment before the start of the project. In the case of schools 7-10, this was less than three months and the 
average was adjusted to take account of this. 


2. School champions were asked to evaluate the pilot project through responding to items on a 
Likert scale (see Appendix 5G). We received 13 responses (three from one school). All agreed or 
strongly agreed with the statement ‘Il was given enough guidance and information to be able to 
effectively run film clubs’. Eleven of thirteen agreed or strongly agreed with the statements ‘Teachers 
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at my school positively engaged with film clubs’ and ‘Film clubs are a sustainable approach to whole- 
school professional development’ (one missing, one unsure). Only one school champion disagreed 
with the statement ‘Using IRIS Connect has been good value for money’. 


3. It is reasonable to conclude that the intervention is feasible if schools and participants say 
they intend to continue using the technology through film clubs. All bar one of the school champions 
indicated at the end of the project that they intended to continue using IRIS Connect, and most 
explicitly stated that they intended to continue using or extending the film club format. The one 
negative school champion response came from the school which sent in three end-of-project school 
champion evaluations. The other two responses were more positive, so two out of the three 
responses from this school indicated that the school would continue to use IRIS Connect.. 


4. The intervention requires a school champion to spend time liaising with the headteacher 
regarding the integration of the project into the school’s improvement strategy. The school champion 
also had to spend time organising and preparing for film clubs. School champions found it difficult to 
quantify just how much time they had spent on these activities since organising meetings and liaising 
with the headteacher were woven into their regular duties. Participating teachers were expected to 
prepare for, and participate in, each of film clubs one to three (estimated average time per teacher 1.5 
hours per film club). They were also expected to collaborate in lesson planning and reviews for film 
clubs four and five, and one teacher was expected to select clips from the lesson to provide a focus 
for discussion. Each of film clubs four and five demanded about two hours of teachers’ time plus the 
time taken for editing, which could be up to two hours. Assuming a group of four teachers in a group, 
this would entail an average of half an hour editing time. The final film club lasted up to an hour and 
did not require preparation. Therefore, each participating teacher was expected to devote about ten 
hours to the project. 


5. School champions were split 50/50 between agreeing and disagreeing with the statement ‘I 
had to spend a lot of time to organise and set-up film clubs’, but only one referred to ‘preparation time’ 
in response to an open ended question about what been time consuming for them (see Appendix 5G). 
Only just over 5% of participants completing the end-of-project survey disagreed with the statement 
‘The time | have spent using IRIS Connect has been worth it’, although 14% agreed with the 
statement ‘The amount of preparation needed for film clubs has been too much to manage’. 


6. Several school champions suggested the challenges for teachers’ time could be addressed if 
schools were able to schedule film clubs into their professional development planning for the year. 
Some schools had incorporated some use of IRIS Connect into their professional development 
planning before agreeing to take part in the pilot. Teachers in these schools faced less of a challenge 
to their time management. Schools which scheduled film clubs in addition to their previously planned 
activities presented teachers with a greater challenge. Although the intervention does present schools 
and teachers with some challenging demands on time, the testimony of the participants in this pilot is 
that these demands can be met. 

The other main limitation on the feasibility of the intervention is teachers’ fear of being observed. In 
the baseline survey, the overwhelming majority of participants and non-participants asserted that they 
‘can trust their colleagues to be supportive if they observe my teaching’ (see Table A5F.3). 
Nonetheless, only about one third disagreed with the statement ‘I feel under pressure if colleagues 
observe my teaching’. By the end of project survey, the proportion of participants and non-participants 
declaring that they felt under pressure during observation had declined, but remained at 33% for 
participants and 40% for non-participants. The intervention appeared to have made a difference, but 
this kind of change takes time to become embedded in a school’s culture and this issue should be 
borne in mind in the design of any efficacy study. 
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Cost 


The cost of the intervention may be evaluated in different ways. Each of the schools participating in 
the project had either bought a licence to use IRIS Connect technology before participating in the 
project or they had already committed to buying one. Therefore, one way of judging the cost of the 
intervention is to only take account of the additional costs which were incurred as a direct 
consequence of the project. However, since a licence was a precondition of participation, it also 
makes sense to calculate the cost of the equipment. 


The cost of the IRIS Connect technology 


Schools who buy a licence for the IRIS Connect technology receive training on how to use the 
package and how they might embed its use in their plans for school improvement. They also receive 
on-demand telephone support for technical problems. These services are provided as part of the 
contract. 


The cost per teacher depends on the number of participants in the school which, in the pilot, ranged 
between 4 and 20, with an average of 9. Table 7 summarises the technology and licence costs for a 
medium to large primary school with an average number of teachers (nine). Small primary schools are 
charged a platform licence at half the rate quoted in Table 7. Schools commit to a three-year licence. 
The annual costs in Table 7 are based on a third of the three-year platform licence and a third of the 
one-off cost of purchasing the video technology (in other words, the cost of one year of the pilot). The 
table also shows the cost implications of the choice of video technology. 


Table 7: Annual cost of hardware and platform licence for a primary school 


Hardware Discoveryt LiveViewt 
School size Medium/Large Small Medium/Large Small 
Platform Licence £1,998.33 £998.33 £1,998.33 £998.33 
Video Technology £566.67 £566.67 £998.33 £998.33 
Total Hardware and Licence £2,565.00 £1,565.00 £2,996.66 £1,996.67 
Cost per teacher (average 9 teachers) £285.00 £173.89 £332.96 £221.85 
Cost per child £11.40 £6.96 £13.32 £8.87 


t The ‘Liveview’ system offers the possibility of remote (off-site) in-ear coaching which was not utilised in this intervention, 
but some schools opt for this system. 


Additional project costs incurred 


Each school was allocated a budget of £1,000 to cover the costs of travel to project meetings and of 
providing supply cover for teachers attending those meetings. So the expected cost of travel and 
supply was roughly £120 for each teacher participating in the project. Since there is some evidence of 
spillover effects on teachers in the schools who were not participating in the project, it is also worth 
noting that cost per teacher (participants and non-participants) in the schools was just over £50. 


As noted earlier, school champions were not able to identify how much time they had spent on the 
intervention. On the basis of feedback provided during and at the end of the project we have made a 
very rough estimate of between 25 hours (£660) and 50 hours (£1,320). The intervention required a 
time commitment from classroom teachers of about ten hours. Given an average FTE salary for 
primary school teachers of £33,400 (DfE, 2016b) and a working year of 1,265 hours, this translates 
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into a per-teacher cost of £264. In the majority of schools in the pilot, film clubs were arranged in 
place of other allocations of time by the school (for example, as an alternative to other forms of 
professional development). However, it is important to note that school champions and participants 
devoted time to the intervention which could have been devoted to other activities. 


Participants’ views about value for money 


Only 5% of teachers responding to the end-of-project survey disagreed with the statement ‘The time | 
have spent using IRIS Connect has been worth it’. At the end of the project, 9 out of 13 school 
champions either agreed or strongly agreed with the statement ‘Using IRIS Connect has been good 
value for money’. Only one school champion disagreed. Although school champions reported that the 
intervention had been time consuming for them, they also believed that using IRIS Connect 
represented good value for money (for details see Appendix 5G). 
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Conclusion 
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1. The overwhelming majority of teachers believed that the intervention was a good use of their 
time and had improved their practice. 


2. During the seven-month pilot, three of the eleven participating schools chose not to engage with 
the second half of the project in which teachers videoed their own lessons. 


3. The intervention demanded a substantial proportion of teachers’ development time. A further trial 
should provide schools with sufficient time to build this into their annual plans and to embed the 
intervention in their plans for school improvement. 


Formative findings 


We believe there are several ways in which the intervention might be refined in order to (1) increase 
engagement by schools and teachers who are less confident with using video technology to prompt 
discussion of teaching and learning, (2) focus thinking on critical differences between ways of leading 
classroom dialogue and providing feedback, and (3) encourage teachers’ discussions to focus on 
reasons why teachers’ actions are helping children to learn (as opposed to focusing on what they do 
and do not like). The following should be considered: 


Recruitment for an efficacy trial should take place sufficiently early to allow schools to 
schedule films clubs as a core element of their strategy for professional development and 
school improvement. This will reduce the likelihood of teachers perceiving film clubs as a 
difficult-to-sustain addition to workload, and should at least partially address the issue of low 
summer term participation identified by the pilot. 

The intervention seems to operate in two ways, (1) by helping teachers to identify possible 
improvements in practice and to discuss the realisation of principles in teaching, and (2) by 
helping teachers to become more comfortable with discussing their classroom work with 
others. The second of these processes takes longer to embed and this carries implication for 
the length of an efficacy trial. 

The online guidance should be strengthened to emphasise the value of open dialogue 
between teachers who discuss implications for their own practice. This might be achieved by 
adding a couple of short clips from film clubs exemplifying, for example, differences between 
approaches to film club leadership. This might also encourage teachers to debate and reflect 
on their own practices relating to key issues before moving on to the next question. 

It would be helpful to emphasise (and exemplify through a video clip) the value of discussion 
which focuses on why a particular example of teaching might be engaging pupils in a 
particular way that is fostering learning (this might be contrasted with a video clip where 
teachers are effectively saying ‘I like that’ or ‘I’m not so sure about that’). 

It would also be helpful to highlight the way in which the online platform can be used to 
identify and comment on moments in a lesson which exemplify an aspect of teaching deemed 
critical to learning, and encourage sharing discussion of these moments between schools. 
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Interpretation 


This pilot has evaluated an intervention using IRIS Connect technology as a package that 
incorporates several elements in a presumed causal chain. The advantage of an intervention like this, 
which changes the way that teachers think about their work and which develops their capabilities, is 
that the benefits should be long-lasting since they take the form of an ‘investment’. This means that 
relatively small changes in the achievement of pupils in any one year could justify an intervention on 
the basis that longer-term benefits will ensue. But this raises two obvious challenges for further 
evaluation: (1) securing a sample that would have power to detect relatively small changes in pupil 
achievement, and (2) identifying the relative importance of each step in the logic chain. 


The overwhelming majority of participants believed that the intervention had affected their practice 
and we found sufficient supporting evidence to accept this as a reasonable working interpretation. 
The intervention has a clear focus on the relationship between task design, classroom dialogue, and 
feedback. This focus was understood by the schools and the teachers. On the basis of a range of 
evidence from videos of lessons, interviews, and ‘before and after’ surveys, this evaluation has found: 


e moderate evidence of change in school climate; 
e strong evidence that film clubs promote discussion of teaching and learning; 


e moderate evidence of change in teachers’ thinking which can be attributed to the intervention; 
and 


e moderate evidence of changes in practice which can be attributed to the intervention. 
Future evaluation and issues for an efficacy trial 


We believe that the survey items used in this pilot evaluation to gather evidence of teachers’ thinking 
about feedback have proved useful in detecting variation relevant to the aims of the project. The 
potential of the video technology for providing evidence about practice could be exploited by (a) 
asking teachers to video a lesson and comment on one of their own lessons before the intervention 
(as a reference point that they can use and, if they are willing, share with others at some later point, 
and (b) using one camera to focus on a small group of pupils so that their thinking and learning can 
be tracked through a lesson. 


In terms of moving the learning on, we believe two options should be considered when thinking about 
future research and publications. First, since this evaluation concentrated on change in teachers’ 
thinking and practice it would make sense to investigate whether changes of this nature and scale 
lead to improvements in outcomes for children. 


Second, the logic chain in this intervention shares some steps with other interventions which do not 
necessarily involve the use of video technology and which do not necessarily involve a focus on 
feedback. IRIS Connect is fully aware of the importance for the intervention of securing ‘whole-school’ 
commitment and the challenges that must be faced in securing this. Its collaboration with Whole 
Education aims to address these challenges. Issues with whole-school commitment are illustrated by 
the relatively low engagement by three schools in the creation of videos of teachers’ lessons despite 
full engagement in the first part of the intervention. 


The design of a further trial would need to consider carefully the nature of any comparator group. 
There are several issues to be addressed: 
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If eligibility to participate in an RCT is restricted to schools that do not yet have video 
technology there is threat to generalisability since schools which already have video 
technology are likely to have school managements and perhaps school cultures which are 
more ready to make constructive use of it. If schools which already have video technology 
are allowed to participate then it would be important to make sure that the trial is balanced on 
this school characteristic (not only in terms of having video technology, but the extent of prior 
experience within the school). 

Since the intervention depends on the availability of video equipment in the school, the 
design of the trial would need to be clear as to whether the evaluation was of ‘state purchase 
of video equipment for schools’ or an evaluation of the effectiveness of schools choosing to 
spend their resources on video equipment. Given the widespread policy stance towards 
school autonomy it is difficult to envisage circumstances in which video equipment might be 
purchased centrally and distributed to schools. Moreover, it would not be possible to 
generalisable from the effectiveness of free distribution of equipment to schools which 
choose to purchase the equipment, given the relationships between the motivation and 
beliefs of school leadership teams, school culture, and the way in which video equipment is 
likely to be used in schools. If, on the other hand, the evaluation is of the effectiveness of 
school choice of spending on video equipment then there will be a problem of recruiting 
sufficient schools. For a large-scale trial, there will never be a sufficient number of schools 
making their first purchase of a video technology system. This might be addressed simply by 
giving schools that agree to participate the option of spending the money on video 
technology and then comparing those who do with those who do not. Given that the sample 
of schools that might be included in this kind of evaluation have not already chosen to use 
video technology, it is likely that the proportion of schools in the evaluation who chose to 
spend the money on video technology would be quite small, making this a very expensive 
option. Dictating that a random half of the sample would be required to buy video technology 
whilst the other half could buy whatever they wanted as long as it was not video technology 
also creates a comparison problem (according to the widely asserted benefits of autonomy). 
Moreover, since IRIS Connect have now ‘rolled out’ their film club model with the focus on 
dialogic teaching and teacher feedback as part of their general offer to schools, it will be not 
be possible to randomise schools which have IRIS Connect into an intervention group using 
the film club model with a focus on teacher feedback, and a control group which has no 
access to this intervention. 


Therefore, we believe that an efficacy trial would prove more useful if it focused on evaluating a 
specific part of the logic chain presented in Figure 1. For example, an efficacy trial could focus on one 
of the following claims: 


that using IRIS Connect technology enables schools to increase the level and depth of 
teachers’ collaboration in reviewing teaching and learning in their lessons; 

that using IRIS Connect improves teacher learning by offering a means to review lessons, 
enabling teachers to re-visit and compare instances of teaching and learning; or 

that a film club format starting with carefully chosen extracts from other lesson videos and 
moving on to teachers reviewing their own lessons will provide a more cost effective way of 
improving teachers’ thinking and practice in some specific way (for example, by fostering 
feedback) than either (a) a film club with an (external) expert facilitator (as reported in the 
literature), or (b) a lesson study format in which the focus of lesson planning and review is 
chosen by the participating teachers. 
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Appendices 


Appendix 1: Examples of Information and Consent Forms 

1A Memorandum of understanding with participating schools 

1B Information and opt out form for parents 

1C Information and Consent Form for Videoing of Teacher Discussion of Lesson Video 
1D Information and Consent Form for discussion of videoed lessons online (Teachers) 
1E Information and Consent Form for interviewees (Teachers) 


Appendix 2: Information about the intervention 

2A Guidance for film clubs 

Appendix 3: Data collection instruments 

3A Baseline survey administered in January 

3B End point survey administered in July 

3C (School) Champion survey administered in July 

3D Information from school Project Leaders gathered at meeting held April 22"? 
3E Protocol for telephone interviews with teachers 


Appendix 4: Examples of data collection 
4A Champion report of a ‘whole-school’ film club 


4B Summary of champions’ comments in focus group meeting 22d April 2016 


Appendix 5: Results tables by source of evidence 
5A IRIS metrics data 

5B School champion midpoint focus group 

5C School champion midpoint feedback 

5D Film club organisation and practice 

5E Videos of lessons shared by teachers 

5F Teacher baseline and end of project surveys 

5G School champion end of project survey 

5H Teacher interviews 
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Appendix 1A Memorandum of understanding with 
participating schools 


Overview of your commitment 


The Pilot Phase of the Education Endowment Foundation-funded research project on how IRIS- 
Connect can support higher-order professional learning will take place between January 1° and July 
318 2016. In choosing to participate in the Pilot Phase your school commits to: 


(i) Nominating a project lead from your school to act as the key contact for this work 


(ii) The school project lead working with the project team to embed the pilot into your 
school's professional development programme. This will involve: 


a. attending two learning events in London 


b. making use of the IRIS-Connect platform’s facilities for (a) written and shared 
reflections and (b) forms for reviewing teaching, supported by the project team 


c. encouraging teachers to use IRIS-Connect to focus on practice and rationale for 
teachers’ feedback to children in the classroom, supported by the project team 
(iii) Providing access to the evaluation team 


Your commitment to the evaluation 


The evaluation of the Pilot Phase involves gathering evidence that will help the EEF to judge whether 
the intervention is ready to be evaluated through a full randomised controlled trial. In order to make 
this judgement it is necessary to collect some indicative evidence about the effects of using IRIS- 
Connect on teachers’ thinking and practice, namely on formative feedback. Two types of evidence (A 
and B below) will be collected from every school. Two types of evidence (C and D will be collected 
from some schools). By agreeing to participate in the trial the school commits itself to providing A and 
B and indicates it will be willing to participate in ether C or D if approached to do so. 


The School undertakes to arrange the following: 


A) Pro Formas 


(i) To complete and return a pro forma that will summarise (i) the ways in which IRIS- 
Connect has been used in the school between January 2016 and June 2016; (ii) the way 


in which these uses of IRIS-Connect are related to the school’s professional development 
and school improvement strategies; and (iii) the validity and practicability of ways of 
measuring children’s learning. 

(ii) To arrange for each teacher in the school to complete short pro formas in January 
and July 2016 regarding (i) their use of IRIS-Connect; (ii) their beliefs about the role of 
teachers’ feedback to children in children’s learning and (iii) their use of feedback in 
teaching and (iv) ways that children use feedback. 


B Sharing examples of videos of lessons 


(iii) To share (with the evaluation team) anonymised' versions videos of extracts from 


at least three lessons and any teachers’ reflections added through the online platform. 
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(iv) To arrange for teachers who share videos of their lessons (and their reflections on 
lessons) to sign a consent form which will be returned to the evaluation team. 

(v) To send a letter informing parents about the evaluation and providing them with an 
opportunity to opt their child out of video footage that will be shared with the evaluation 
team. 


C Telephone conversations (some schools will be approached to arrange two telephone 
conversations) 


(vi) To be willing to arrange telephone conversations between two teachers (including 
the school project lead) and a member of the evaluation team. The evaluation team will 


contact three of the participating schools for conversations about (i) the implications of 
using IRIS-Connect for teachers’ workload and value for money and (ii) the extent to 
which using IRIS-Connect has led to changes in teachers’ thinking and practice. 


D Face-to-face interviews with teachers (some schools may be approached to arrange 
interviews with 2-3 teachers about their use of IRIS) 


(vii) To be willing to arrange face-to-face conversations between two to three teachers 
(including the school project lead) and a member of the evaluation team. The evaluation 


team will contact two of the participating schools for conversations about (i) the 
implications of using IRIS-Connect for teachers’ workload and value for money and (ii) the 
extent to which using IRIS-Connect has led to changes in teachers’ thinking and practice. 


Signed: Date 
(Name, Role, Organization) 


Signed: + 
(Douglas Archibald, Director, Whole Education) Date: 7 December 2015 
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Appendix 1B Information and opt out form for parents 


Dear Parent 


As part of its commitment to continually improving its practice, your child’s school is taking part in an 
evaluation of the IRIS-Connect system for videoing lessons. This evaluation has been commissioned 
by the Education Endowment Fund which has been set up by the government to identify effective 
approaches to teaching. 


The purpose of recording some lessons is to help teachers to improve their practice by carefully 
analysing the teaching and learning that has taken place. This evaluation will gather evidence to test 
how much difference the system makes to the quality of teaching. 


The evaluation will be conducted by three researchers at the University of Birmingham who will view 
some video recordings of lessons and discuss these with the teachers. The IRIS system allows 
teachers to blur the image of people in the classroom so that individual children cannot be identified. 
Also, there will be no transfer of files or individual images across the Internet. The researchers will log 
on to a ‘remote’ facility to view already ‘blurred’ videos which they will not be able to download. 


If you are unwilling for a ‘blurred’ video of a lesson in which your child has taken part to be viewed by 
a member of the research team please sign and return the slip at the bottom of the letter. Please 
return this form before (enter appropriate date here) to the school secretary or directly 
to the leader of the research evaluation team: 

Professor Peter Davies 

School of Education 

University of Birmingham 

Edgbaston 

B15 2TT 


Yours sincerely 
Peter Davies 


Evaluation of IRIS Connect 


1am NOT willing for an anonymised video recording of a lesson in which my child has taken part to be 
viewed by a researcher at the University of Birmingham. 
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Appendix 1C Information and Consent Form for Videoing 
of Teacher Discussion of Lesson Video 


The University of Birmingham is evaluating the use of IRIS Technology by schools. As part of that 
evaluation we ask for your permission for the three members of the evaluation team to view a video 
recording of your discussion of a lesson recording.. 


This evaluation has been commissioned by the Education Endowment Fund. The evaluation is being 
carried by the University of Birmingham and the leader of the evaluation team is Professor Peter 
Davies. 


No school or individual will be named in the files we use to store the data or any subsequent 
document. We will allocate a code number for each school and each video of teachers discussing a 
lesson. The recording will be retained in the IRIS-Technology secure system so it can be copied or 
sent to others through the Internet. Access to the folder will be strictly controlled under the guidance 
of Professor Peter Davies who is leading the project. 


If at any time you wish to terminate the recording of the discussion please indicate this and the 
recording will be stopped. You may also request any record of the discussion to be deleted. 


If you have any questions about the research please contact Professor Peter Davies at the University 
of Birmingham. His telephone number is 0121 414 4820 and his email is p.davies.1@bham.ac.uk. 


If you are willing for the discussion to be recorded using the IRIS-Connect system please sign below 
to indicate that you have read and agree to participate under these conditions. 


Thank you 


| have read and understood the information. | agree to the discussion being videoed and for this video 
to be used in the evaluation of the use of IRIS technology. 
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Appendix 1D Information and Consent Form for discussion 
of videoed lessons online (Teachers) 


The University of Birmingham is evaluating the use of IRIS Technology by schools. As part of that 
evaluation we would like to discuss your use of the video technology using the IRIS platform. This will 
involve discussion about the lesson, your use of video in the lesson and how the recording of the 
lesson has been used. 


This evaluation has been commissioned by the Education Endowment Fund. The evaluation is being 
carried by the University of Birmingham and the leader of the evaluation team is Professor Peter 
Davies. 


No school or individual will be named in the files we use to store the data or any subsequent 
document. We will allocate a code number for each school and each interviewee. All recordings will 
be kept secure with files saved in a password protected folder. Access to the folder will be strictly 
controlled under the guidance of Professor Peter Davies who is leading the project. 


If at any time you wish to terminate the discussion please indicate this and the interview will stop. You 
may also request any record of the discussion to be deleted. 


If you have any questions about the research please contact Professor Peter Davies at the University 
of Birmingham. His telephone number is 0121 414 4820 and his email is p.davies.1@bham.ac.uk. 


If you are willing to take part in this online discussion please sign below to indicate that you have read 
and agree to participate under these conditions. 


Thank you 


| have read and understood the information about the purpose of this interview. | agree to participate 
in this interview and for my answers to be used in the evaluation of the use of IRIS technology. 
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Appendix 1E Information and Consent Form for 
interviewees (Teachers) 


This interview is being conducted to help with the evaluation of the use of IRIS technology by schools. 
The interviews will be used to find out how schools are using the IRIS technology and the reasons for 
using the technology in these ways. This evaluation has been commissioned by the Education 
Endowment Fund. The evaluation is being carried by the University of Birmingham and the leader of 
the evaluation team is Professor Peter Davies. 


No school or individual will be named in the files we use to store the data or any subsequent 
document. We will allocate a code number for each school and each interviewee. All recordings will 
be kept secure with files saved in a password protected folder. Access to the folder will be strictly 
controlled under the guidance of Professor Peter Davies who is leading the project. 


If at any time you wish to terminate the interview please indicate this and the interview will stop. You 
may also request any recording to be wiped clean. 


If you have any questions about the research please contact Professor Peter Davies at the University 
of Birmingham. His telephone number is 0121 414 4820 and his email is p.davies.1@bham.ac.uk. 


If you are willing for to be interviewed and for the interview to be transcribed and an anonymous 
record retained in a secure folder at the University of Birmingham please sign below to indicate that 
you have read and agree to participate under these conditions. 


Thank you 


| have read and understood the information about the purpose of this interview. | agree to participate 
in this interview and for my answers to be used in the evaluation of the use of IRIS technology. 
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Appendix 2A Guidance for film clubs 


Film clubs 1-3: Teachers viewing and discussing video clips provided through the online platform 


Guidance for each of film clubs 1-3 was provided online (through the ‘group’ function in online 
platform) under the following sub-headings: 


(i) introduction; 

(ii) Classroom talk & (Film Club 1) teaching; (Film club 2) questioning and group talk; (Film 
club 3) feedback: one ‘page of guidance for each module supplemented by 1-2 ‘talking 
heads’ videos of 30 sec — 1 min 30 second duration. 


(ili) Things to consider: more note-form guidance including a section ‘what sort of questions 
should | be asking myself when watching lesson video clips?’ 

(iv) Pre-film club reflection: more guidance on the topic 

(v) Film club;: guidance (linked to two 1-2 minute video clips) on how to organise the film club 
meeting — 


1 Ensure everyone is aware of the lesson context 

2 Watch the clip once without pausing 

3 Consider the following questions: 

What do you think about the nature of the True or False task? 
What other ways could this maths question have been presented? 
How Is the task introduced? 

How does the task affect the nature of the discussion? 

4 Watch the clip a second time with these questions in mind. 

5 Watch the clip a third time. Feel free to pause and discuss where you feel it 
appropriate. Use the time-stamped comments feature to make a note of anything that 
stands out or that you find interesting (you may notice comments from other teachers 
involved in the project; you may wish to consider these when having your own 
discussions). 

(vi) Report your film club findings; 

Champions were encouraged to record their film club sessions and to share these online 

with the evaluation team. Alternatively, they were given a set of open response items 

through which they could summarise the film club. For example, after film club 2 the 

prompts asked about: attendees; the main points of discussion during ‘your own 

reflections’; the main points of discussion after viewing the film clips; any changes in 

perspective during the course of the discussion; the intentions of the project team in 

future practice following the film club; judgement about the effectiveness of the film club. 
(vii) Reflections: prompts on how to review own practice following the film club. 


Film clubs 4 & 5: Teachers reviewing and discussing videos of their own lessons 


Teachers were encouraged to organise their fourth and fifth film clubs (in which they used their own 
lesson or lessons) in six steps: 


1 Collaborative planning: in which they were asked to plan together a ‘rich task or question’. Four 
short video clips were provided as exemplification. The planning guidance also included a ‘reminder 
about dialogue’ (with a list of 11 desirable characteristics of dialogue) and they were also reminded 
about feedback with a short clip featuring Dylan Wiliam and Table A2A.1 which was adapted from 
Hattie & Timperley’s (2007) review of research on feedback. 
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Table A2A.1 (which was untitled in the online guidance) 


Feedback about 
the self 


What a superstar! You did that really well. Weak 


Good when 
supported by 
strategies for 
learners to try 


So you're saying the two sums would not be equal; can 
you work out what the final equation would read that 
proves that? 


Feedback about 
the task 


By suggesting something might go wrong you make the 


peeahacs pout reader worry about your character, which creates 


strategies or 
processes 
needed in tasks 


Powerful in the short 
term 


suspense and makes them want to read on. Is there any 
way you could introduce this earlier to engage the reader 
right from the beginning? 


So you adjusted the weight at the front of your model 
plane, after you found it kept nose-diving, by removing a 
Feedback about ||paperclip. Is there anyway you can make it glide for|/ Powerful in the long 
self -regulation longer? Where can you find out how to improve this? Has term 
anyone managed to make theirs go further? What can you 
learn from theirs? 


2 Reflecting, editing, commenting and sharing: in which they were again encouraged to focus on 
episodes in the lesson which they found interesting in terms of dialogic teaching and feedback. 
Instructions were provided on how to edit lessons to focus on the episodes and how these could be 
shared with other teachers in the group through the online platform. 


3 Peer feedback: in which teachers were invited to share their lesson with one other colleague who 
would add comments to the video clip using the online platform. This invitation was accompanied by 
guidance that comments should “(i) respect and support the teachers and pupils in the video; (ii) 
suspend judgement to avoid faulty assumptions; (iii) focus on interactions and how they are linked; 
and (iv) look deeper and seek context”. 


4 Sharing with the project group: in which the video clip(s) and associated comments would be 
shared with the whole project group and comments invited from all group members. 


5 Pre film club reflections: in which group participants were invited to view the clips and to reflect on 
aspects of the teaching and learning they wanted to discuss at the film club. 


6 Film club: in which the group met to discuss the video clip(s) with the following focus: “(i) the 
effectiveness of the planned task/question; (ii) how learners are invited into dialogue; (iii) how 
dialogue develops; (iv) how language promotes a positive learning culture; (v) what pupil responses 
tell you about their learning; (vi) the source of feedback (teacher, self, task, peer) and how it is 
offered; (vii) how feedback relates to the perceived learning intention; (viii) how feedback promotes 
independent learning? (ix) how feedback creates thinking and furthers understanding; (x) how 
learning is led (pupil-led / teacher-led) and the impact this has”. 
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Film Club 6 Evaluating the experience 


In Film Club 6 teachers were asked to reflect on their experience using the following questions and 
champions were asked to summarise the thoughts expressed using the online platform. The 
questions were: 


1 Please give an overview of your school approached the project, including who was involved and 


how this was organised. 

2 Did you like this approach to professional learning? 

3 Why? 

4 What did you and the other participants learn? 

5 What changes, if any, have been made to classroom practice? 

6 What benefits for learners did you observe? Do you have any other evidence to back this up? 
7 Were there any negative effects? 

8 What challenges did you experience? How did you overcome these? 


9 Where do you plan to go from here? 
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Appendix 3A Feedback in Teaching and Experience with 
IRIS-Connect: A (baseline) survey 


What is this short survey for? 


Your school is taking part in a project to evaluate the use of IRIS-Connect Technology. The project is 
funded by the Education Endowment Fund. The evaluation is being conducted by researchers from 
the University of Birmingham. Teachers' use of feedback has been chosen as a focus by which to 
measure the impact of IRIS-Connect Technology on teaching. 


We would be grateful if you would complete this survey and seal it in an envelope we have provided 
and return it to your school office who will post responses to the University of Birmingham. We have 
asked for your name so that we can match your answers to this survey with any other information you 
provide during the course of the evaluation. By returning your survey in the envelope provided your 
answers will not be disclosed to anyone else in the school either now or in the future. 


All data will be stored on a secure password protected folder at the University of Birmingham. 


Evaluation results will NOT BE DISCLOSED IN RELATION TO NAMED SCHOOLS OR 
INDIVIDUALS at any stage to maintain confidentiality. 


For further information, email Professor Peter Davies p.davies. 1@bham.ac.uk 


Please sign below to confirm that you have read this information and that you are willing for 
your answers to be used in the evaluation and matched with any other information you choose 
to disclose on the condition that your responses are treated as confidential. 


Name 


Signature 


School 


Date 


Thank You! 
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You are asked to complete a grid below to indicate how you think about feedback in your teaching. 
You may use feedback in different ways at different times and you will be asked about these 


differences after the grid. When completing the grid please think about what the way in which you use 


feedback most frequently with the average or typical child you teach. 


Please tick one place on the 54321012345 scale in each row to indicate where you would position 
yourself in relation to the two statements. Ticking O indicates that you position yourself as neither 
leaning towards the statement on the left nor leaning towards the statement on the right. If you 
change your mind, simply cross out your first mark and tick another place on the scale. 


My feedback typically 
provides the child with 
a simple next step they 

can easily achieve 


My feedback 
typically presents 
the child with a 
serious challenge 


My feedback carefully 
highlights the 
differences between 
alternative ways of 
solving a problem 


Feedback typically 
gives students a 
correct way of 
solving a problem 


My feedback typically 
makes plain the 
sequence of steps a 
child should follow in 
solving a problem 


My feedback 
typically 
concentrates on 
helping a child to 
review the steps they 
have taken to solve a 


problem. 
My feedback typically my ipedbacs 
typically leaves the 
leaves the child feeling 
child to judge 
good about what they 
whether they have 
have done. 
done a good job 
My feedback generally My feedback 
reflects my judgement typically tells the 
about why a child has child how close their 
given a particular answer was to my 
answer. answer. 
My feedback 
My feedback typically ; ; 
tells a child what the plea) nape a ae 


problem is with their 
thinking 
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My feedback 


My feedback typically concentrates on 


concentrates on helping 


children to understand |[_ ]/L JL JIL LILI EE I making sure that 


; children know what 
different ways of seeing ce areca 4h 
a problem. y P 


do. 


8. The previous question asked you to characterise how you think about feedback in 
general. However, you may use feedback in different ways in different contexts. 


If you think this is the case, please use the space below to give details of any of your 
responses above which you think are particularly context dependent: 


Professional Development 


(Please tick one column for each statement) 


9. To what extent do you agree Strongly 
with each of these statements? Disagree 


Strongly 


Disagree | Unsure | Agree Agree 


I can trust my colleagues to be 
supportive if they observe my 
teaching 


I learn a lot from my colleagues 


I feel under pressure if colleagues 
discuss my teaching 


I find it more useful to reflect on 
my lessons on my own 


The final section of this questionnaire (overleaf) is designed for teachers who have used IRIS- 
Connect. 
If you have NOT used IRIS-Connect, please stop at this point. Thank you for completing this survey. 
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Professional Development Using IRIS-Connect 


(Please tick one column for each statement) 


10a. To what extent do you 
agree with each of these 
statements? 


Strongly 
Disagree 


Disagree 


Unsure 


Agree 


Strongly 
Agree 


It is easy to use IRIS-Connect 
cameras to record videos 


It is easy to use the IRIS-Connect 
platform to view my recorded 
videos 


10b. To what extent do you 
agree with each of these 
statements? 


Strongly 
Disagree 


Disagree 


Unsure 


Agree 


Strongly 
Agree 


Using IRIS-Connect has helped me 
to improve my understanding of 
my teaching 


Using IRIS-Connect has improved 
my understanding of children’s 
needs 


Using IRIS-Connect has made me 
feel more vulnerable. 


The amount of preparation needed 
for film clubs has been too much to 


The time I have spent using IRIS- 
Connect has been worth it 


manage 


Using IRIS-Connect has become an 
important part of my professional 
development practice 


The focus on formative feedback 
has fit within my own professional 
development needs 


11. How often have you used IRIS- 
Connect in each of these ways? 


Never 


Once or 
twice 


occasions 


3-4 


5or 
more 
occasions 


In-ear coaching (being coached) 


In-ear coaching (as a coach) 
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1-1 mentoring (being mentored) 


1-1 mentoring (as a mentor) 


In a group viewing and discussing an 
example of practice from another 


school 


In a group viewing and discussing an 
example from a member of the group 
In a group sharing in the planning, 

teaching and review ofa lesson (as in 


Lesson Study) 


12. To what extent have Participated | Participated | Participated 
you used IRIS-Connect Not but not and modest as major 
in each of these participated | important part of my part of my 
contexts? to my job job job 
Supporting trainee 

teachers 

Support for NQTs 


As part of the school’s 
appraisal system 


As part of the school’s 
collaborative CPD 
programme 


As part of a collaboration 
between schools in our 
partnership 


13. Has IRIS-Connect helped you make changes to your classroom practice? 
Please give details/examples: 
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14. Have you changed how you think about feedback as a result of using IRIS-Connect? 
Please give details/examples: 


15. Have you had a particular focus/purpose with regards to your use of IRIS Connect? 


16. Would you continue to use IRIS-Connect in the future if the option was available to 
you? 
If yes - would you use it in the same way as now? If no - why not? 


If you have any further or other comments, please continue overleaf. 


Thank you for completing this survey. 
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What is this short survey for? 


Your school is taking part in a project to evaluate the use of IRIS-Connect Technology. The project is 
funded by the Education Endowment Fund. The evaluation is being conducted by researchers from 
the University of Birmingham. Teachers' use of feedback has been chosen as a focus by which to 
measure the impact of IRIS-Connect Technology on teaching. 


We would be grateful if you would complete this survey and seal it in an envelope we have provided 
and return it to your school office who will post responses to the University of Birmingham. We have 
asked for your name so that we can match your answers to this survey with any other information you 
provide during the course of the evaluation. By returning your survey in the envelope provided your 
answers will not be disclosed to anyone else in the school either now or in the future. 


All data will be stored on a secure password protected folder at the University of Birmingham. 


Evaluation results will NOT BE DISCLOSED IN RELATION TO NAMED SCHOOLS OR 
INDIVIDUALS at any stage to maintain confidentiality. 


For further information, email Professor Peter Davies p.davies. 1@bham.ac.uk 


Please sign below to confirm that you have read this information and that you are willing for 
your answers to be used in the evaluation and matched with any other information you choose 
to disclose on the condition that your responses are treated as confidential. 


Name 


Signature 


School 
Date 


Thank You! 
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UNIVERSITY0OF 
BIRMINGHAM 


My views on Feedback in Teaching 


You are asked to complete a grid below to indicate how you think about feedback in your teaching. 
You may use feedback in different ways at different times and you will be asked about these 
differences after the grid. When completing the grid please think about what the way in which you use 
feedback most frequently with the average or typical child you teach. 


1. Please tick one place on the 54321012345 scale in each row to indicate where you would 
position yourself in relation to the two statements. Ticking 0 indicates that you position yourself as 
neither leaning towards the statement on the left nor leaning towards the statement on the right. If 
you change your mind, simply cross out your first mark and tick another place on the scale. 


Bur oe 25. PO BZ 33-4 |S 
My feedback typically My feedback 
provides the child with typically presents 
a simple next step they OJOJO)O)O/OO)O)O)0)0 the child with a 
can easily achieve serious challenge 
My feedback carefully 
Feedback t ll 
meee tae oi ce a 
differences between |[_]/L JL JIL LILI ELLIE Sere 
: correct way of 
alternative ways of 
solving a problem 
solving a problem 
My feedback 
My feedback typically typically 
makes plain the concentrates on 
sequence ofstepsa |L]/LI/LI/LI/LI/LIILI/ LILI L/L] helping a child to 
child should follow in review the steps they 
solving a problem have taken to solve a 
problem. 

My feedback typically my ipedbaek 
leaves the child feelin play Ces tie 
¥ OJOJO|OJOO|OJOO|O]O]  chita to juage 

good about what they 
whether they have 
have done. ; 
done a good job 
My feedback generally My feedback 
reflects my judgement typically tells the 
about why achildhas |{_]/L]/LJ)/LJ/LJ/LJ/L/ L/L /L | LJ] child how close their 
given a particular answer was to my 
answer. answer. 
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My feedback 
typically helps a child 
tells a child what th 
pce ee : LOO) UL) ||} Ly) to work out what the 
problem is with their an 

thinking problem is with their 


My feedback typically 


thinking. 
My feedback 
concentrates on 


My feedback typically 
concentrates on helping 
children to understand |[_]/L JIL JIL LIE EI I 
different ways of seeing 

a problem. 


making sure that 
children know what 
they are expected to 
do. 


2. The previous question asked you to characterise how you think about feedback in 
general. However, you may use feedback in different ways in different contexts. 


If you think this is the case, please use the space below to give details of any of your 
responses above which you think are particularly context dependent: 


Professional Development 


(Please tick one column for each statement) 


Strongly 
Agree 


3. To what extent do you agree Strongly 


: : Disagree | Unsure | Agree 
with each of these statements? Disagree 6 8 


I can trust my colleagues to be 
supportive if they observe my 


teaching 


I learn a lot from my colleagues 
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I feel under pressure if colleagues 
discuss my teaching 


I find it more useful to reflect on 
my lessons on my own 


The final section of this questionnaire (overleaf) is designed for teachers who have used 
: IRIS-Connect. : 
: If you have NOT used IRIS-Connect, please stop at this point. Thank you for completing this : 
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Professional Development Using IRIS-Connect 


(Please tick one column for each statement) 


4a. To what extent do you agree 
with each of these statements? 


Strongly 
Disagree 


Disagree 


Unsure 


Agree 


Strongly 
Agree 


It is easy to use IRIS-Connect 
cameras to record videos 


It is easy to use the IRIS-Connect 
platform to view my recorded 
videos 


4b. To what extent do you agree 
with each of these statements? 


Strongly 
Disagree 


Disagree 


Unsure 


Agree 


Strongly 
Agree 


Using IRIS-Connect has helped me 
to improve my understanding of 
my teaching 


Using IRIS-Connect has improved 
my understanding of children’s 
needs 


Using IRIS-Connect has made me 
feel more vulnerable. 

The amount of preparation needed 
for film clubs has been too much to 


The time I have spent using IRIS- 
Connect has been worth it 


manage 


Using IRIS-Connect has become an 
important part of my professional 
development practice 


The focus on formative feedback 
has fit within my own professional 
development needs 


Attending film clubs has made me 
more likely to reflect on my own 
practice 


Film clubs have led to increased 
teacher collaboration in my school 


Film clubs have been an effective 
approach to professional learning 


5. How often have you used IRIS- 
Connect in each of these ways? 


Never 


Once or 
twice 


occasions 


3-4 


5or 
more 
occasions 


In-ear coaching (being coached) 


In-ear coaching (as a coach) 


1-1 mentoring (being mentored) 
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1-1 mentoring (as a mentor) 


In a group viewing and discussing an 
example of practice from another 
school 

In a group viewing and discussing an 
example from a member of the group 


In a group sharing in the planning, 
teaching and review ofa lesson (as in 
Lesson Study) 
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6. To what extent have Participated | Participated | Participated 
you used IRIS-Connect Not but not and modest as major 
in each of these participated | important part of my part of my 
contexts? to my job job job 
Supporting trainee 

teachers 

Support for NQTs 


As part of the school’s 
appraisal system 


As part of the school’s 
collaborative CPD 
programme 


As part of a collaboration 
between schools in our 
partnership 


7. Has IRIS-Connect helped you make changes to your classroom practice? 


Please give details/examples: 


8. Have you changed how you think about feedback as a result of using IRIS-Connect? 
Please give details/examples: 


Education Endowment Foundation 


58 


IRIS Connect 


9. Have you had a particular focus/purpose with regards to your use of IRIS Connect? 


10. Would you continue to use IRIS-Connect in the future if the option was available to 


you? 
If yes - would you use it in the same way as now? If no - why not? 


If you have any further or other comments, please continue overleaf. 


Thank you for completing this survey. 
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Appendix 3C (School) Champion survey administered in 
July 


What is this short survey for? 


Your school is taking part in a project to evaluate the use of IRIS-Connect Technology. The project is 
funded by the Education Endowment Fund. The evaluation is being conducted by researchers from 
the University of Birmingham. 


As part of the evaluation, researchers must collect evidence on the costs of using IRIS-Connect in 
terms of time, money and resources. As a result, we would like to hear from all school project leaders 
about the costs to their school and them personally of setting up and running a programme based on 
IRIS-Connect. 


We would be grateful if you would complete this survey and seal it in an envelope we have provided 
and return it to your school office who will post responses to the University of Birmingham. We have 
asked for your name so that we can match your answers to this survey with any other information you 
provide during the course of the evaluation. By returning your survey in the envelope provided your 
answers will not be disclosed to anyone else in the school either now or in the future. 


All data will be stored on a secure password protected folder at the University of Birmingham. 


Evaluation results will NOT BE DISCLOSED IN RELATION TO NAMED SCHOOLS OR 
INDIVIDUALS at any stage to maintain confidentiality. 


For further information, email Professor Peter Davies p.davies.1@bham.ac.uk 


Please sign below to confirm that you have read this information and that you are willing for 
your answers to be used in the evaluation and matched with any other information you choose 
to disclose on the condition that your responses are treated as confidential. 


Name 


Signature 


School 


Date 


Thank you! 
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Using IRIS-Connect for Professional Development 
1. To what extent do you agree Strongly | Disagree | Unsure | Agree | Strongly 
with each of these statements? Disagree Agree 


Observing others is an important 
part of my school’s approach to 
professional development 


Discussing teaching and learning is 
an important part of my school’s 
approach to professional 
development 


My school was ready to make good 
use of IRIS-Connect when we first 
bought it 


Staff at my school were initially 
enthusiastic about the prospect of 
using IRIS-Connect 


Getting some other members of 
staff started on IRIS-Connect was 
difficult 


Please use this space to give any further details on your answers: 


2. Why has your school invested in IRIS-Connect? 


3. Have you had a particular focus/purpose with regards your use of IRIS Connect? 


Education Endowment Foundation 61 


IRIS Connect 


Education Endowment Foundation 


62 


Section 2 


IRIS Connect 


Film Clubs as a Whole-School Approach to Professional Development 


4. To what extent do you agree 
with each of these statements? 


Strongly 
Disagree 


Disagree 


Unsure 


Agree 


Strongly 
Agree 


Film clubs are an effective way of 
running whole-school CPD 

Film clubs fit within the 
professional development 
approach of my school 


The focus on feedback was suitable 
for my school’s needs 


Using IRIS-Connect has increased 
the amount of collaborative 
professional learning taking place 
at my school 


I was given enough guidance and 
information to be able to 
effectively run film clubs 


The online content on the IRIS- 
Connect platform to support film 
clubs was stimulating 


Teachers at my school positively 
engaged with the film clubs 


Please use this space to give any further details on your answers: 


5. What aspect(s) of your film clubs do you think was particularly effective? 


6. How would you change film clubs for any future use? 
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Section 3 


Costs of Running Film Clubs 


7. To what extent do you agree 
with each of these statements? 


Strongly 
Disagree 


Disagree 


Unsure 


Agree 


Strongly 
Agree 


I had to spend a lot of time to 
organise and set-up film clubs 


After the first film club, organising 
subsequent film clubs was much 
quicker 


Using IRIS-Connect has been good 
value-for money 


Film clubs are a sustainable 
approach to whole-school 
professional development 


The success of film clubs is highly 
dependent on the project leader’s 
input 


Please use this space to give any further details on your answers: 


8. What have been the most time-consuming aspects of organising film clubs? 


9. Has your school incurred any substantial costs other than the IRIS-Connect subscription fee? 
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Section 4 of 4 
The Effectiveness of Video-Based Professional Development 


10. Would you continue to use IRIS-Connect in the future if the option was available to you? 
- If yes - would you use it in the same way as now? 
-  Ifno, please explain why not. 


11. What are the problems with using videos for professional development? 


12. Have you noticed any clear changes in the practice or attitudes of teachers in your school? 


If you have any further comments, please continue overleaf. 


Thank you for taking the time to complete this survey. 
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Appendix 3D Information from school Project Leaders 
gathered at meeting held April 224 


1. How many teachers (not including support- 
staff) are there at your school? 


2. How many teachers have been actively involved 
in your school’s film clubs? 


3. Why were the participating teachers chosen to be involved? 
(ie. are you running this as a whole-school project? Or have you selected particular 
teachers and, if so, on what basis?) 


4. Have you received any parental out-out forms? 


Yes No 


5. If yes, how are you dealing with the affected classes? (tick one or more) 


No videos are being shared from these classes 

Affected children are being positioned off-camera 
Affected children are being edited out of the video 
Affected children are being relocated out of the class during filming 
Other 
If other, give details: .......... 0.0... cece cece cece cece eee ee ee ee eee eeeeeeeeaeeeeeeeeaes 


6. How many film clubs have you held so far? 


1 2 
Name Name 
Year Year 
GTOUP ns. .csiiesscvaisteaatts GTOUD gas ccedscctissrnstelascas 
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Email: Email: 


NB: 1-2 teachers are needed per school, of which the project lead can be one but not the only one. 
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Appendix 3E: Focus group with school champions April 
224 (mid-point of pilot) 


Questions About the Set-Up 

Q1- Did you have enough time to set-up the project? 

- Will you have completed all film clubs by the end of the academic year? 

- Ina possible future evaluation, a measure of student achievement would be used. It 
is likely a baseline would be needed. Do you think this could have been fitted in 
during the set-up? 

Q2 - What were the largest tasks when initially setting up the project? 

- (Possible tasks): Registration on the IRIS platform, learning to physically set up and 
use the cameras, accessing and watching videos on the IRIS platform, 
training/supporting other staff, working the intervention into wide school CPD 
plans, planning/preparation for film clubs (initial/ongoing). 

- Were there any problems with that? 

- Roughly how long did you need to spend doing that? 

Q3 - What are the major tasks in continuing the project now the initial set-up is 

complete? 

-__ Roughly how long do you spend on that? 

Q4 - Have there been any costs to your school other than the IRIS-Connect 

subscription and your time? 

- (If costs are mentioned, the project lead survey can get figures on them) 

- Were these costs foreseen? 


Questions About Initial Impressions of the Film Clubs 
Q5 - Have you been clear on how to organise film clubs and what to do in them? 
- What else would you have like to know/been given? 
- Has the content on the IRIS platform been helpful? 
Q6 - Has the content on the IRIS-Platform about feedback and the opening film 
clubs been relevant/useful/stimulating? 
- What else would you have like to have been given? 
- Has the focus on feedback and content matched your school’s profession 
development needs? 
Q7 - How much have you had to direct/structure your film clubs? 
- Have you had a specific focus for each session? 
- Have you asked staff to prepare or follow-up anything? 
Q8 - Have your staff positively engaged with film club? 
- Why were they enthusiastic/reluctant? 
- Did you have to do anything to get everyone involved? 
- Have your staff done the amount of preparation needed/expected? 
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Appendix 3F: Protocol for telephone interviews with 
teachers 


Q1. How familiar are you with using video to reflect on teaching? 
- What are the main things that you have changed? 
- Which video can we see that in? 
- What prompted you to make that change? 
- Did you have a specific reason for using video for that/in this lesson? 


Q2. Do you think the videos show any changes in your classroom practice? 
- What are the main things that you have changed? 
- Were you happy with the feedback in that lesson? 
- Which video can we see that in? 
- What prompted you to make that change? Did you have a specific reason for 
using video for that? 


Q3. Do you think your views on effective feedback have changed during the 
process? 
- Why do you think doing it that way would be more effective? 
- What did you think about [that] before the process? What was/is your typical 
approach? 
- (Can you explain that a bit more? 
- _What prompted you to rethink [that]? 


Q4. Has anything changed for the pupils in your classroom during the project? 
- (Clarify) Will they have noticed differences in your feedback? Or have they 
changed their role in giving or receiving feedback? 
- Can you give me an example of that? 
- What did you need to do to bring that change about? 


Q5. Are there any aspects of your feedback you are still looking to improve? 
- Howdo you think you can go doing that? 
- What further support do you think you need with this? 
- What benefits do you think this change will bring? 


Q5. Has using the video prompted you to rethink any other aspects of your 
teaching? 
- Did you have a specific reason for using video for that? 


Q6. Is there anything you would have changed about the process and how you 
used IRIS? 

- What did you make of the content provided? 

- How useful were film clubs? 

- How would you change your approach if you continued to use IRIS? 


Further questions will be prompted by the reflections the teacher has posted online 
already: what prompted these reflections and how have they affected subsequent 
teaching? 


Education Endowment Foundation 


71 


IRIS Connect 


Questions specifically for this teacher: 


End of questions - Is there anything you wish to add or ask? 
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Appendix 4A: Champion report of a ‘whole-school’ film 


club 


Today's date 


02/02/2016 


School 


Name of Primary School 


Your name 


Champion’s name 


Film Club attendees 


23 names listed 


What were the main 
points of discussion 
following the first video 
clip? 


Whether the lack of modelling or visual aids hindered the task, or 
expanded it and gave a greater opportunity for discussion and 
exploration without making the answer too obvious. 

How the nature of the true or false statement stimulated discussion 
without giving any clue as to the correct response. 

The thinking time the teacher gave the children was seen as a positive, 
as well as the requirement to make the children justify their choices. 
Quotes: 

"No modelling of the task so LA might struggle to give a reason for their 
answer." 

Reply "I agree, but this could narrow the task." 

"We liked the task because it expected them to give a reason for their 
idea - but there was a definite answer. Test understanding - as if they 
didnt really think about it they might say it was true. Putting it in a real-life 
context was good. Thinking time and then talking time." 

"Hopefully it will deepen the level of discussion later. The discussion will 
be child-led depending on their reasons. It allows the teacher to flag up 
mis-conceptions depending on children's answers." 

"The thinking time / action was effective to avoid children shouting out 
the answer and copying their response." 

"Like True/ False tasks, gets away from the children being right’ wrong. 
Clear setting out of the importance of giving a reason for answers." 


What were the main 
points of discussion 
following the second 
video clip? 


The 'excitement' over the disagreement, providing a positive 
environment and a safe place in which children would feel safe to give 
their opinions. The poker face of the teacher gave away nothing as to 
the actual answer. 

The use of language leading to an impression that it was a discussion, 
not an argument e.g. "defend" and "support" not "right" and "Wrong". 
The patience and withholding of judgement by the teacher also 
developed the children's confidence. 

The time given to each side of the discussion stopped children from 
interrupting each other. 

Quotes: 

“Exciting' we have a disagreement! She makes it a positive point that 
the children disagree, but nor does she create an adversarial 
environment between the two viewpoints, this implies a class where it is 
‘safe’ to be right or wrong when investigating a question." 

Reply: "Definitely agree with this, important to value everyone's options 
and reasoning." 

"No indication either way about whether the ch were right or wrong so all 
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ch were happy to explain their thinking." 

"Teacher used phrases that showed it was a discussion/debate and not 
an argument. She used positive language such "defend" and "support" 
rather than "right" or "wrong". She encouraged chn to support each other 
by building on their reasoning." 

"There was no judgement of the children's views, no facial expression so 
the children seemed confident to express their views. Management was 
also good as other children allowed speakers to finish without 'chipping 
in’." 

"Use of 'defend', and 'talk about' rather than ‘what they think' means 
children find it easier to give an answer they are not sure about. She 
implies she is keen to hear their reasoning, not to find our what's 'right' 
initially." 

“Teacher modelled good listening skills and allowed chn time to talk and 
think without rushing them. Chn then mirrored this behaviour and gave 
their peers time to talk." 

"Language used by the teacher to get children's opinions is less 
intimidating than 'can you explain your answer" 

"Lots of time given to fully explain their reasoning and no comments from 
the teacher to break their flow or questions or encouragement that may 
have taken them off task" 


Were there any 
changes in perspective 
during the course of 
discussions? 


There was a move away from introducing a problem with lots of visual 
aids, no letting the children form their own ideas first to simulate 
discussion. 

People also moved away from correcting misconceptions faster, and 
allowing children the space and discussion time to explore and work 
through misconceptions. 


What do the project 
participants want to 
reflect upon in their 
own practice following 
Film Club? 


Giving children the space to be wrong. 
Using less judgemental language. 
Allowing more thinking time. 


Broadly, how effective 
do you feel Film Club 
was as a learning 
experience? 


Very effective 
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Appendix 4B: Summary of champions’ comments in focus group meeting 22d April 2016 


Understanding your Film Club Content Design of Intervention 
How Level of 3 Alm 
: : : e nse 
io a When in sauder " tea you Right Sane ny on oe Useful? L.e.: ou Urdchibs (000 ba a changes 
People| facilitated? choose teachers?| amount? ' (too muchitoo | muchitoo & 
day? | advance? level? ? ? : ss practice? 
? little) little) 
Amount of 
content 3 Alm 
ae eee but ate aes pa all Asics sae Eee 
| “leac" Monday planned | Chosen by asking nee and well Structure foieiaee Level of guidance — if it practice were 
discussion and and teachers to join - all Pitch was | explained,with | wes clear | y was good, but some instant - the key is| 
1 8 3:30-5pm| .. ; . : teachers : linked ...can ; were a : 
refocus where : timetabled} given information |. excellent. appropriate and : staff found it was to monitor that 
ish. ict will always : see evidence of . .__ {Struggle to) : 
necessary. 2-3 weeks and invited. expert effective. : too time exhausting.) .. .__| they continue on 
eee struggle to ee ee each in the fit them in el ies 
‘ find time in a others. a short i 
advance of term. 
Clubs. 
| watched 
/ clips and 3 film Had no 
prepared Aries Clubs wes| expectations re 
Lead ‘crib’ sheet fine - first | time but expected 
2 20 lle meeting cs A\l staff involved!! | much (but Yep! Yep! Yep! Yep! Just right. ee eh : 
phase leaders. acces new for me to all great!) one term | positive impact on 
eel. | follow with ene which was| practice. Happene 
* | key Qs on too small! d instantly. 
it. 
Started 80% | read Module 2 | Pitch was 3 film 
an a. aaa Headteacher ieee however They tere deat Slade meee a 
a. oe a ee ee eee sie = structu ee iain oo pinided a fei 
confortable to " asked staff] beinvolved, | modules! | vocab | yar. ne ere a ‘| the Spring 
open to read thought | needed i : termto 
discussions their parts - were the | recapping complete 
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themselves. produced right with staff. themin 
summary amount. wes a 
of each Content in struggle. 
module for each 
them to module 
take away. excellent. 
Content 
excellent, 
| took on materials 
role of pre- were well 
Open learning chosen. 
discussions - and Pre- cerca 3 film 
split into 3 feedback ir leaming res icra Staff took alot | Guidance in terms | clubs 
groups. Guided a staff | feeding | Bachna |: Se framicach: || Set hindubining- Saree ee 
Staff : Staff given option : ____|Club. Outcome in : : seen almost 
4| 20-25 by 1 staff neSinos meeting inte part Was quite | appropriate} ied BEA Yes. session and and outcomes | appropriat jivmeciaicly (each 
member asking (CPD). time : eS GEE focus of would have been | e amount week’. 
Qs- usually staff Staff then consuming ee feedback useful. useful. for our 
: Ss ica during film club? 
sharing own did flipped for (module 2). school. 
practice. learning teachers - 
before film although 
club. could see 
importance} 
Facilitator asain 
looked | Pathfinder group irae 
cay | trough | — selected for ae scbusariis 
Dan led and | ring 1| Motetias. jenthusiasm tried tg Pal " eS ce 
5} 25 cae hr Newing alain i Land 3, Nee Yes. Yes. nature of About right. ons Straight away. 
comments on TT wih Selection. iss pain less. SiGe: right. 
aS x3. Pa Hees 2. Classroom 
questions. | compatible groups. paler 
Food! supports quality 
discussion. 
| facilitated End of | | looked | identified core | Have onl Really hel! 3 film Had different 
a (ies Starting/focusin| School | through | group of teachers | done x NG Mey ten: vey: Reet Veyides: Clubs too | expectations for 
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g discussions |day- extra} content | who | knew would | modules - and discussion. many to fit] different teachers. 
and then first. be happy and content in short | Sawchanges in 
stepping back Emailed [driven to participate §=amount half term - practice 
until | need to reminders | - those with growth] great for hence 3rd} immediately - 
move forward- of what mind-set and M1 and one some surface 
teachers love to} needed to| strong drive to slightly scheduled) level but some 
talk! be done | develop. Used as | more for for next | now embedded in 
and by | part of developing |M2. As non week. 3 for} practice. Have 
when. outstanding class a'unit’ just also seen 
teachers/excellenc| based | right. development of 
e in teaching. had more learning 
time to culture/attitudes 
view. Did with children. 
offer 
teachers 
release 
time- none 
took me up 
on it. 
| led the Worked 
discussion with with 
the assistant Assistant |__ Meeting with 
head. Some Head to Bon eae 
Thursday : discussed who had Maybe too] Pretty quickly- 
staff are more : plan ideal : 
peer sees evenings - eee used IRIS before Good level of many to fit) have already 
7 9 aire usually fnaene and who would be guidance and into the begun to see 
start at willing - we wanted structure. Spring changes in 
teen 4:15. Pee a good range of term. practice. 
question and flexible and 
‘pull’ the move some pens 
information from dates , 
the group. around. 
Facilitated by Planned | Confident with new Too many 
myself. Informal] During | withina | ideas - flexible and film clubs Immediate 
8a 3 _ |Jandfreetotalk.}| School | short HT. | open to changes. within the changes to 
Confortable day. Longer | Confortable and time practice. 
with group. than respectful of each frame. 
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expected. other. Unclear of 
time 
needed for 
each film 
Club. 
Film clips 3 film 
Planned in were right Yes - well Clubs 
fad a short half amount bout} explained. An difficult to | Changes can be 
Reali bye or term - brought uy overview plan in implemented 
and led by Dan| possible fate Chose teachers ee of P rape Very interesting, eyes al 
eH but open due to is 9 a who had flexible Pitched at ; Hs Structured] very useful. Guidance clear ; ee 
discussion with} student | “| schedules and |_. “| good level. iy Se well. | Generated alot} once started. : mies a 
aloe ‘ Planned Maile ee Didn't know would take would Sasa eSon discussion} Ideas can be 
ie taki around mg Rae how long it have been useful : Sand _|taken into account 
Oy ‘ is release would all (to plan release needed in planning. 
Classes. es ; : 
availability. take to plan time). time to 
it in. follow up. 
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1. How many 
People? 


8, 20, 4, 20-25, 25, 6, 8, 3 

This suggests two distinct approaches: Whole school (3 schools) 
and Small Groups/phases (5 schools) 

With the exception of 1 group (chosen by the head), the groups 
were volunteers and therefore taking part on the basis of their 
enthusiasm /availability to do the film clubs. 


2. Who and how 
facilitated? 


In general, there was some leadership from the project lead but 
their role seemed more one of refocusing and facilitating. Staff 
seemed happy to be involved so the group facilitator needed to 
keep the discussion on track rather than to get it moving. One 
school that broke into smaller group tended to have 1 member of 
staff leading each group - usually the one who is sharing their own 
practice. 


3. When in School 
day? 


1 School (8) held the film clubs in an afternoon as they were 
released due to student teachers taking class. This was a small 
group (3). Where all staff were participating (see Q1) the sessions 
were put as a staff meeting. All other were after school, in addition 
to all other commitments. 


4. How was it 
planned in 
advance? 


This question was interpreted in different ways. Several talked 
about timetabling, that it was planned around release time or that 
it was timetabled 2-3 weeks in advance. Others discussed 
preparation for the session. In general, the person leading the 
group pre-read materials and watched clips, making notes ready 
for the session. It was also shared with staff prior to the meeting 
with an expectation that they will also have looked through the 
materials. 


5. How did you 
choose 
teachers? 


This is linked with Q1. Sometimes involvement was decided by 
SLT, sometimes it was entirely voluntary. It would have been 
more useful to hear why schools took one approach over the 
other. There is a suggestion in several responses that the school 
wanted to have ‘pathfinders’ to get things working first or that 
only selected teachers would/could benefit from it. 


6. Right amount? 


Most responses suggested that the content amount was about 
right but maybe a bit too much for module 2. There was wider 
discussion on the day about the role of the project leader in 
condensing the information and whether this was appropriate and 
whether module 2 in particular gave teachers too much to read. 


7. Pitched at the 


Yes 


right level? 

8. Well Yes - one respondent suggested that an overview of/guidance 
explained/clear | about timings would have been useful. Another comment about 
2 this in question 11 too. 

9. Logical Yes 
Structure? 


10. Useful? Le.: 
feedback/focus. 


All comments were positive. Content was interesting, well linked, 
useful - generated a lot of discussion. 
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11. Level of 
guidance/struc 
ture (too 
much/too 
little) 


Most say about right but some staff found it a bit too much. 
Guidance about timings would have been useful (someone 
mentioned that they did not know how long to put aside for film 
clubs or to suggest preparing for it on the day). 


12.3 Film clubs 
(too much/too 
little) 


Some found it about right but there were concerns it was a bit too 
much to fit in to the time frame in some cases. Further concerns 
about guidance over timings are raised. The film clubs raised a lot 
of discussion points/things to do and needed to be spread out. The 
short Spring term made this very difficult to complete. Some 
schools thought this was OK though. 


13. Time to 
changes in 
teacher 
practice? 


All apart from one said that changes happened straight away. One 
said it depends on the individual. There is also an indication in a 
couple of responses that there may be more superficial changes 
initially or that it will take time to see whether the changes are 
embedded. 
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Appendix 5A IRIS metrics data 


The IRIS-CONNECT video platform automatically collated two kinds of information. Figures A5A1-11 
present the number of times someone from a school accessed the content provided on the online 
platform. This included the video clips from other sources provided as inputs for film clubs 1-3. We 
therefore expected more hits during the first three months of the intervention when schools were 
expected to organise film clubs 1-3. Table A5A.1 presents data on the number of videos created by 
teachers at each school during each of three time periods: (i) from when the school first joined IRIS- 
CONNECT to the start of the project. For schools 6-11 this was between 0 and 4 months; (ii) the first 
three months of the intervention (film clubs 1-3); (iii) the final 3 months of the intervention when 
teachers were expected to create videos of their own lessons (film clubs 4-6). 


Platform Use by Week by School - Page and Reflection Video Access Metrics 


Key 


Hits (separate use) on online platform per project participant 


=s==eS = Hits on online platform per teacher in the school 


(in Schools 4, 8 and 10 included all teachers in the project so there is no separate line for ‘per teacher 
in the school) 


Figure A5A.1: Platform Use, School 1 
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Figure A5A.2: Platform Use, School 2 
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Figure A5A.3: Platform Use, School 3 
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Figure A5A.4: Platform Use, School 4 
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Figure A5A.5: Platform Use, School 5 
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Figure A5A.6: Platform Use, School 6 
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Figure A5A.7: Platform Use, School 7 
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Figure A5A.9: Platform Use, School 9 
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Figure A5A.10: Platform Use, School 10 
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Figure A5A.11: Platform Use, School 11 
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Table A5A.1 Video Creation by Schools Before and During the Project 


IRIS Connect 


Before Spring Summer 
First Weer Number of Number of Duration of User Number Duration Wer Number Duration of 
School Recorded Count videos videos per 3 videos Count of videos | of videos Court of videos videos 

Use created months created created created created created 
1 2/11 27 225 12 231:28:34 12 30 26:44:36 7 30 53:15:16 
2 1/15 2 2 1 1:05:38 2 5 1:06:52 0 0 0:00:00 
3 7/14 15 75 13 32:30:35 7 15 8:44:22 5 25 9:23:10 
4 8/14 28 179 34 80:01:45 16 39 16:27:47 17 41 9:49:15 
5 2/15 6 13 5 2:26:45 5 8 1:44:53 0 0 0:00:00 
6 9/15 13 28 21 12:42:14 9 22 10:24:05 16 99 30:58:02 
7 9/15 8 14 11 3:54:54 1 7 2:20:53 4 6 3:29:43 
8 10/15 6 10 10 3:05:03 5 10 8:13:37 11 12 9:08:51 
9 10/15 6 10 10 2:16:31 6 10 7:01:27 0 0 0:00:00 
10 11/15 7 12 18 6:04:28 15 60 26:56:18 8 8 5:03:57 
11 1/16 0 0 0 0:00:00 5 15 2:24:03 1 4 3:27:03 
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Appendix 5B: School champion midpoint focus group 


This summary highlights the observations offered and issues raised by school champions attending 
the project mid-point meeting. 


Integrating the intervention with the schools’ strategy for professional development: The intervention 
creates challenges for the nature and implementation of a school’s strategy for school improvement 
and professional development (CPD). On a practical level it creates demands on teachers’ time which 
are easier to accommodate if a school is able to plan for this demand well in advance (ideally in the 
previous school year). Several school champions commented that the intervention was competing for 
CPD time with other activities that had already been planned. Teachers’ time was used in editing as 
well as discussing the classroom videos. On a strategic level the intervention encourages schools to 
adopt a particular approach to school improvement involving open dialogue between teachers about 
their classroom practice. School champions believed that teachers’ engagement with the intervention 
and the prospects for change were framed by the extent to which the school-level ambitions of the 
intervention were consistent with the aims of the school’s senior leaders. 


Time to embed. School champions commented that it took teachers several weeks to get used to the 
technical and procedural requirements of the system (using the cameras, the online platform and the 
ways these facilities could be used). This adjustment included training teaching assistants to set up 
cameras and learning how to handle transitions at the start and end of lessons. However, most 
concern about ‘time to embed’ related to overcoming teachers’ fears about being filmed and a 
perceived threat that the videos would be used to make negative judgements about their teaching. 


The film club format. Some school champions believed that the extent to which teachers prepared for 
the film clubs was a major factor in the value of the dialogue during film clubs. There was general 
agreement that the sequence of film clubs viewing videos from elsewhere followed by film clubs of 
teachers’ own lessons worked well. One school champion suggested that it would have been useful if 
teachers had filmed one of their own lessons before the intervention so that this could be used as a 
base-line for later reflection. 


Costs. It was noted that the cost to the school depended on which specification of the technology was 
provided through the school’s contract with IRIS-CONNECT. Schools adopted different approaches to 
teachers’ time. A minority released teachers during normal lesson-time whilst the majority held film 
clubs outside of normal teaching time. The critical factor here is whether the intervention replaces 
other forms of CPD or is in addition to other CPD in which case there is an additional cost borne by 
teachers. 


Effectiveness: School champions believed that the intervention was making a difference to teachers’ 
willingness to discuss their classroom and to teachers’ thinking about their practice. 
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Appendix 5C: School champion midpoint feedback 


During the project mid-point meeting, IRIS-CONNECT asked school champions to answer 13 
questions by writing their responses on post-it notes. These responses are summarised in Table 
A5C.1. 


Table A5C.1 Summary of school champion’s written responses to questions posed by IRI- 
CONNECT at the project mid-point 


14. How many 8, 20, 4, 20-25, 25, 6, 8, 3 

People? This suggests two distinct approaches: Whole school (3 schools) and Small 
Groups/phases (5 schools) 
With the exception of 1 group (chosen by the head), the groups were volunteers 
and therefore taking part on the basis of their enthusiasm/availability to do the 
film clubs. 

15. Who and how In general, there was some leadership from the project lead but their role 
facilitated? seemed more one of refocusing and facilitating. Staff seemed happy to be 

involved so the group facilitator needed to keep the discussion on track rather 
than to get it moving. One school that broke into smaller group tended to have 1 
member of staff leading each group — usually the one who is sharing their own 
practice. 

16. When in School | 1 School (8) held the film clubs in an afternoon as they were released due to 
day? student teachers taking class. This was a small group (3). Where all staff were 

participating (See Q1) the sessions were put as a staff meeting. All other were 
after school, in addition to all other commitments. 

17. How was it This question was interpreted in different ways. Several talked about timetabling, 
planned in that it was planned around release time or that it was timetabled 2-3 weeks in 
advance? advance. Others discussed preparation for the session. In general, the person 

leading the group pre-read materials and watched clips, making notes ready for 
the session. It was also shared with staff prior to the meeting with an expectation 
that they will also have looked through the materials. 

18. How did you This is linked with Q1. Sometimes involvement was decided by SLT, sometimes 
choose it was entirely voluntary. It would have been more useful to hear why schools 
teachers? took one approach over the other. There is a suggestion in several responses 

that the school wanted to have ‘pathfinders’ to get things working first or that 
only selected teachers would/could benefit from it. 

19. Right amount? | Most responses suggested that the content amount was about right but maybe a 
bit too much for module 2. There was wider discussion on the day about the role 
of the project leader in condensing the information and whether this was 
appropriate and whether module 2 in particular gave teachers too much to read. 

20. Pitched atthe | Yes 
right level? 

21. Well Yes — one respondent suggested that an overview of/guidance about timings 
explained/clear ) would have been useful. Another comment about this in question 11 too. 

2 
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22. Logical Yes 
Structure? 

23. Useful? l.e.: All comments were positive. Content was interesting, well linked, useful — 
feedback/focus. | generated a lot of discussion. 

24. Level of Most reported that the level of guidance was about right but some staff found it a 
guidance/struct | bit too much. Guidance about timings would have been useful (someone 
ure (too mentioned that they did not know how long to put aside for film clubs or to 
much/too little) | suggest preparing for it on the day). 

25. 3 Film clubs Some found it about right but there were concerns it was a bit too much to fit in 
(too much/too to the time frame in some cases. Further concerns about guidance over timings 
little) are raised. The film clubs raised a lot of discussion points/things to do and 

needed to be spread out. The short Spring term made this very difficult to 
complete. Some schools thought this was OK though. 

26. Time to All apart from one said that changes happened straight away. One said it 
changes in depends on the individual. There is also an indication in a couple of responses 
teacher that there may be more superficial changes initially or that it will take time to see 
practice? whether the changes are embedded. 
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Appendix 5D: Film club organisation and practice 


How did schools organise participation in film clubs? 


Three schools (4, 8 and 10), arranged whole school participation in film clubs. In one of these schools 
(10) this was reported as voluntary (see Table A5D.1). The remaining schools adopted a small group 
approach. In three of these cases participation was voluntary. There was no obvious difference 
between the form of organisation chosen by those schools which had been using IRIS-CONNECT 
before the project and those which were new to using IRIS-CONNECT. 


Table A5D.1: Data on film clubs shared with evaluation team 


School Format Participants go eer a Film club data shared 
1 Selected Group 8 Yes five videos of film clubs and one summary from champion 
2 Volunteer Group 3 Yes two summaries from champion) 
3 Volunteer Group 9 Yes two summaries from champion) 
4 Whole School 17 Yes one summary from champion 
5 Selected Group 5 Yes one summary from champion 
6 Selected Group 5 No two videos of film clubs 
7 Volunteer Group 8 No two videos of film clubs and two summaries from champion) 
8 Whole School 20 No two videos of film clubs and one summary from champion 
9 Selected Group 4 No No data 
10 Whole Seoul 20 No three videos of film pee se ig summaries from 
11 Selected Group 4 NO two summaries from champion) 


What were the topics of discussion in film clubs? 


The feedback forms and film club videos were analysed looking for the main points of discussion 
during the session. They are briefly summarised below: 


Dialogic culture and routine. There was a large amount of discussion of the dialogic culture that 
had been embedded in some of the example videos. Teachers argued that the children shown in 
the videos were very familiar with the approach (e.g. a culture inviting disagreement and children 
explaining their positions in a way that was not ego-involving or threatening). 

Teachers refraining from giving the correct answer ‘too soon’. This was modelled in the first film 
club with the implication that a key aspect of the dialogic approach is to avoid closing down 
discussion through feedback which provides the ‘right answer’. Teachers discussed the value of 
this strategy and also considered arguments for providing information, modelling and directing 
children. 

Tasks for discussion. The relationship between the task presented and the ensuing dialogue was 
a common point of discussion. Teachers debated the characteristics of tasks which promoted 
discussion amongst children. 

Timing, pace and classroom management. Teachers commented on the use of hand signals and 
the pace of tasks. There was discussion of the pace of the lessons which some teachers argued 
was rather slow. There was particular interest in the use of ‘wait time’ after teachers on videos 
had asked questions of the children. 

Questioning: participants discussed the quality of questions and the value of open-ended 
questions and higher-order questions to promote dialogue. They also drew out the specific 
language used for questions, the purposes of questions (e.g. whether they are for the teacher’s 


Education Endowment Foundation 89 


IRIS Connect 


benefit, to promote dialogue or to scaffold pupil understanding). There were also many links 
between the questions and dialogue. 

e Pupil to Pupil Dialogue: Many groups discussed how to promote pupil to pupil dialogue and how 
to get pupils to respond to each other, promoted by some clips that did this well. Some noted that 
they would have liked more group talk and one group pointed out the difficulties of always hearing 
pupil-pupil dialogue using IRIS. 

e Teacher language: a fair amount of discussion identified features of language. Sometimes this 
was related to a specific task in a specific clip (e.g. discussion of the word ‘solid’ linked with one 
clip). Often it was picking out how the feedback was worded, (e.g. “Can you add to that?”) 

e Pupil independence, ownership and engagement: there was some discussion of how to 
encourage children to take ownership of the dialogue, and how to encourage reflection from them. 

e Groups and classroom organisation: part of the difficulty of translating some of what was seen in 
the film clubs was difference in group size. There was some discussion of whether particular 
approaches or tasks would work in smaller/larger groups or with younger children for example. 
There were also both positive and negative points raised about engagement. This included 
discussion of how to bring all children into the discussion but also criticism of some platform clips 
where it was unclear whether all children were engaged. 

e = The role of praise: the smaller role for praise evident in many of the clips was discussed in several 
groups. Some just noted that the praise was task rather than ego involved, one comment made 
by one group was that there was no ego involvement at all and this had taken it too far. 


Adherence, ownership and reflection. In some schools teachers used film clubs as a stimulus for quite 
wide ranging discussion. Teachers in these schools adopted a more relaxed approach throughout and 
appeared, from the video evidence, to be more engaged in discussing the issues and trying to relate 
these to their own practice. This was evident in contributions from teachers which began with phrases 
such as ‘I find it hard...’, ‘Most of our feedback...’, ‘We often...’, ‘My children...’, ‘I say...’, ‘| tend to...’, 
‘| think sometimes we...’, ‘So what I’m trying to do is....’. Discussion largely focused on implementing 
ideas. In these schools, teachers commented on things they had tried and discussed challenges and 
successes, referring in detail to each other’s videos whilst picking out things to do differently. Film 
clubs in other schools appeared to stick rigidly to working through the questions provided in the 
support on the online platform. In this more rigid format the session leader asked questions posed in 
the guidance on the platform. Relatively few opinions were aired, followed by a small amount of 
clarification/agreement before the next question was ‘read out’. In one case, the staff seemed slightly 
reticent to contribute and the sessions were very formulaic. These film clubs were more likely to be 
dominated by one film club member who shared their experience at considerable length. This may 
have been due to reluctance of other teachers to share and a school champion aiming to model the 
kind of disclosure they were trying to encourage in others. : Nonetheless, even in schools which 
started with a rigid format, later film clubs became more relaxed. In the few schools which included all 
their teachers in the intervention we observed one case where film clubs kept rigidly to the questions 
on the online support and one which was characterised by open discussion and application to 
practice. 


Identification of implications for teaching. In some schools, discussion in film clubs focused on 
implementing changes in teachers’ own practice. In these schools, the project leader encouraged 
teachers to commit to actions. Sometimes these commitments were quite general. For example, one 
school champion asked, “How are we going to use this to improve our own dialogue in classrooms?” 
and the group agreed to a couple of points for action. But several schools who took a highly managed 
approach in which teachers committed themselves to specific actions. Teachers in one school 
agreed to specific actions to complete before the next session. The school champion started the 
following session by asking (somewhat humorously), ‘Has everyone done the homework?’ In another 
school the school champion outlined in the opening their expectations about what teachers would do 
during the project. Subsequent sessions in this school showed teachers agreeing to create 2 or more 
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videos to illustrate progression in specific aspects of teaching. Teachers stated specific objectives and 
identified lessons or activities that they would undertake before the next session. 


In other schools, possible points for development were identified in a more tentative and general way: 
the talk was of what we ‘want to develop’ and ‘reflecting on how [they] can best facilitate this.’ 
Discussion in one school identified aspects of teaching to develop yet no specific targets were 
suggested. The project leader closed this session by suggesting that teachers should ‘maybe have a 
think about your questioning as you are in class, such as the questions they have given us’. It is 
possible these schools translated points from film club sessions into changes in classroom practice. 
For other schools, the management of this transfer was far more managed 
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Appendix 5E: Videos of lessons shared by teachers 


Teachers shared some of the videos they created with the evaluation team. These videos provide 
evidence of some of the feedback which teachers provided to children. (see Table A5E.1) Although 
the intervention design encouraged production and sharing of videos of teachers’ practice in the 
second half of the pilot period, some lesson videos were shared with the evaluation team during the 
earlier stage of the project. Forty-three lesson videos were shared. Seven teachers shared more than 
one video with more than a month between the lessons. It was not possible to discern clear changes 
in practice through this very small sample. Therefore, the analysis of the lesson videos is presented 
as a reflection of teachers’ practice during, largely, the second half of the pilot project. 


Twenty-four of the lesson videos were shared by teachers from schools in which film clubs were 
conducted through open, exploratory discussion in which there were strong references to teachers’ 
practice. Nine of the lesson videos were shared by teachers in which film clubs were strongly directed 
by one individual, closely following the online guidance and with more limited reference to teachers’ 
practice. The reporting in this appendix compares the lesson videos from these two types of school. 


Table A5E.1 The number of lesson videos shared by teachers from each school 


School Number of lessons shared with the Number of project teachers in the 
evaluation team school 
1 ‘ 
2 a ae ‘ 
; 7 
: 19 
; rr a) ‘ 
: ee eee 
; a 9 
z 17 
9 ee ae 1 
. ee 16 
u ‘ 
Total 45 


These videos were analysed in terms of the nature of children’s participation in classroom dialogue 
(Table A5E.2) and in terms of the type of feedback provided by teachers (Tables A5E.3). Table A5E.2 
shows that teachers dominated whole class dialogue and that they provided limited opportunities for 
children to talk about their work with each other. Table 5E.3 presents an analysis of the 43 videos 
using the same categories as the survey of teachers. Each video extract was coded on a scale of -5 
to +5 according to how closely the teaching resembled the left hand or right hand statement in each 
row. Table A5E.3 shows the considerable range in feedback visible in the videos. School champions 
argued that this reflected diversity in situation: that one type of feedback might be the most 
appropriate in one circumstance whilst another type of feedback would be more suitable in another 
situation. 
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Table A5E.2 Frequency of type of role for children in classroom dialogue (n=43) 


None/ very 
little 


Some 
examples 


Frequent/ 
complex 


Participation by children in whole class 


dialogue 


Pupil gives extended response 31 8 4 

Pupil asks an unprompted question 39 3 0 

Pupil responds directly to a point made by another pupil 30 10 3 
Child-child dialogue 

Small group discussion 34 2 7 

Sharing with partner 31 8 4 

‘Reciprocal teaching’ style where children were 39 4 0 


encouraged to use feedback strategies with peers 


Table A5E.3 Characterisation of teachers’ feedback visible in shared lesson videos using the 


same format as the teacher survey 


Strength of tendency 
towards left hand 
statement 


Strength of tendency 
towards right hand 
statement 


5 | -4| -3|-2|-1/ 0 


1)/2/;3),4)5 


1 Gives easy/simple next 


Poses serious 


0;2;6);2;/;0/0j,2)10;/11; 9) 1 
steps challenges 

2 Highlights differences Giles caret 

between alternative | Be | a Pa hs Bf eal Sid 

response/method 

responses/method 

3 Gives correct series of olilslslai3ltslalialala4 Reviews pupil's steps or 
steps or points points 

4Makes pupilfeel good | 0 | 1|1| 2) 2/7] 0] 0) 5/18] 7 | eaveste ill . 


5 Reflects the teacher's 
judgement about pupil’s | 0 | 5 
current understanding 


6 Tells a pupil the problem 
with their method/thinking 


10; 9 | 2 | 2 


Tells the pupil how close 
they were to teacher's 
answer 
Helps the pupil to work 
out what the problem is 
with their 
thinking/method 


7 Helps pupils understand 
different ways of seeinga | 0 | 1 | 8 
problem. 


10} 2 | 3 


Helps pupils know what 
they are expected to 
do/say. 


The categories in Table A5F.3 were analysed to compare the 24 videos from schools with more ‘open 
and applied’ film clubs were compared with the 9 videos from schools with ‘closely directed, less 
applied’ film clubs. The lessons from schools using the more ‘open applied’ film clubs were 
categorised as displaying feedback which gave more emphasis to: posing serious challenges; 
highlight different methods; reviewing pupils’ steps; leaving judgements to pupils, reflecting the 
teachers’ judgement of a pupil’s current understanding; helping pupils to work out problems; and 
helping children to see alternative ways of seeing problems. This is a small sample which must be 
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treated with caution, but a simple t-test suggested that each of the differences was statistically 
significant at the 0.05 % level and 5 of the seven were statistically significant at the 1% level. 
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Appendix 5F: Teacher baseline and end of project surveys 


Survey response rates 

Table A5F.1 summarises the response rates to the baseline and end of project surveys. Schools were 
asked to arrange for all teachers in the school to complete each survey, but rates of response from 
non-participants varied between schools. 


Total 
teachers 
Unique in 
School | Baseline Endpoint Both Respondents school 
Project | Not | Total | Project | Not | Total | Project | Not | Total | Project | Not | Total 
1 4 9 13 4 8 12 4 8 12 4 9 13 20 
2 3 8 11 4 21 | 25 3 6 9 4 23 | 27 33 
3 5 2 7 6 13 | 19 4 2 6 7 13 | 20 25 
4 17 0 17 16 0 16 14 0 14 19 0 19 24 
5 4 8 12 4 4 8 4 4 8 4 8 12 13 
6 5 12 | 17 4 6 10 4 5 9 5 13 | 18 21 
7 5 0 5 8 1 9 4 0 4 9 1 10 18 
8 17 2 9 14 1 15 14 1 15 17 2 19 21 
9 1 2 3 1 2 3 1 1 2 1 3 4 13 
10 16 1 17 9 0 9 9 0 9 16 1 17 18 
11 3 8 11 3 5 8 2 5 7 4 8 12 12 
80 52 122 73 61 134 63 32 95 90 81 80 


Table A5F.1 Survey response rates by school 


Teachers’ reporting of their approach to feedback 


Table A4F-2 presents survey results for the questions asking participants to indicate their typical 
approach to each of seven aspects of feedback. Only those teachers (varying from 88 to 95 across 
the 7 items in Table A5F.2) who had participated in both surveys are included in this analysis 
(complete case analysis). There may be unobserved differences between those who completed both 
surveys and those who only took part in one. However, this problem applies to non-participants as 
well as participants. The response scale had 11 divisions in which the furthest to the left indicated 
strong emphasis on the statement on the left. The division furthest to the right indicated strong 
emphasis on the statement on the right. 


There were three differences (significant at .05%, chi-squared test) between project participants and 
non-participants in the baseline survey. These are shown by baseline. Project participants were more 
likely to assert that their feedback ‘made plain the sequence of steps a child should follow in solving a 
problem’, ‘typically tells the child how close their answer was to my answer’ and ‘typically tells a child 
what the problem is with their thinking’. That is, at the time of the baseline survey, teachers included 
in the project were /ess likely than other teachers in their schools to assert that their feedback 
displayed the qualities that the intervention aimed to develop. 


A comparison of project participants’ responses to the final and baseline surveys shows that they 
were more likely to assert their feedback presented children with serious challenges, highlighted 
differences between alternatives, reflected their judgement about the children’s reasoning (as 
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opposed to the correctness of their answer), helped children to identify problems in their reasoning 
and helped children to understand different ways of seeing a problem. Each of these differences is 
consistent with the intentions of the intervention. There were also some significant differences 
between non-participants’ assertions in the baseline and end of project surveys. However, only two of 
these (‘my feedback highlights differences between alternative ways of seeing a problem’ and ‘ my 
feedback concentrates on helping children to understand different ways of seeing a problem’) were in 


the direction encouraged by the intervention. 


Table A5F.2 Teacher Self-Characterisation of Feedback (reduced scale) 


Effect size t 
(change for 
Left iy Right participants 
relative to non- 
participants) 
1. My feedback ___ | Participant 41.7 | 15.0 | 43.3 ; 12 
typically provides the | 8@Seline (on participant 151.6 119.4) 29,0 | MY eedback typically 
as A : a presents the child 
child with a simple Participant 938/190] 571 with aceeriule 
Teast aicie Endpoint 'Non Participant | 59.9] 15.6] 34.4 challenge 
2. My feedback ___ | Participant 40.0 | 20.0 | 40.0 -.03+ 
carefully highlights Baseline Non Participant | 35.5 | 22.6 | 41.9 | Feedback typically 
the differences Participant 54.0 | 20.6 | 25.4. | gives pupils a correct 
between alternative Endpoint | Non Participant way of solving a 
ways of solving a nepal 56.3 | 9.4 | 34.4 problem 
problem 
3. My feedback .__ | Participant 25.0 | 11.7 | 63.3 | My feedback typically 42 
typically makes plain Baseline Non Participant | 9.7 | 32.3 | 58.1 concentrates on 
the sequence of Participant 986 127.0| 44.4 | helping achild to 
steps a child should Endpoint | Non Participant review the steps they 
follow in solving a repo 32.3 | 35.5 | 32.3 | have taken to solvea 
problem problem. 
Participant 57.6 | 15.3 | 27.1 29 
4. My feedback | Baseline |Non Participant | 61.3 | 16.1 | 22.6 |My feedback typically 
typically leaves the leaves the child to 
child feeling good Participant 49.2 | 20.6 | 30.2 | judge whether they 
about what they | Endpoint have done a good 
have done. Non Participant | 74.2 | 9.7 | 16.1 job 
5. My feedback .__ | Participant 39.0 | 39.0 | 22.0 -.48t 
generally reflects my Baseline Non Participant | 65.5 | 20.7 | 13.8 | My feedback typically 
judgement about Participant 58.7 | 23.81 17.5 |_ tells the child how 
why a child has Endpoint | Non Participant close their answer 
given a particular | Endpoint 53.1 | 281) 18.8 | was to my answer. 
answer. 
6. My feedback | paceline | aticipant 20.0 | 3.3 | 76.7 | My feedback typically 36 
typically tells a child |__| Non Participant | 9.7 | 3.2 | 87.1 | helps achild to work 
what the problem is __, | Participant 11.1 | 14.3 | 74.6 | out what the problem 
with their thinking | Endpoint ro participant | 15.6 | 12.5 | 71.9 | is with their thinking, 
7. My feedback Participant 50.0 | 10.0 | 40.0 My feedback 124 
typically Baseline |Non Participant | 51.6 | 16.1 | 32.3 concentrates on 
concentrates on making sure that 
helping children to | Endpoint | Participant 57.1 | 23.8 | 19.0 | children know what 


understand different 
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ways of seeing a Non Participant | 56.3 | 21.9 | 21.9 do. 
problem. 


tT calculated by Cohen’s d 
+ The intervention aimed to increase use of feedback described by the left hand side of this row, so for 
this row a negative effect size is desirable. 


We analysed correlations between responses to the items in Table A5F.2 using the baseline survey. 
Only two correlations were statistically significant after making a Bonferroni adjustment. Items 2 and 7 
were moderately correlated (r=. 46) and item 4 was weakly correlated with item 5 (r=.24). We also 
compared changes in participants’ beliefs about their own feedback in four schools where use of the 
online system tailed off in the summer term and those schools where use of the online platform was 
more or less maintained. Participants in the schools where use of the online system tailed off were 
significantly (at the .05% level) less likely to report that ‘My feedback carefully highlights the differences 
between alternative ways of solving a problem’ and ‘My feedback typically concentrates on helping a child to 
review the steps they have taken to solve a problem’. 


Teachers’ views of collaborative learning 


Table A5F.3 reports teachers’ responses to questions about their attitudes towards working with and 
learning from colleagues. There was one observable difference between participants and non- 
participants in the baseline survey (measured through a chi-squared test with columns amalgamated 
to makes sure there were sufficient entries in each column for this test). Participants were more likely 
than non-participants to assert that ‘I learn a lot from my colleagues’ (p<.01). This probably reflects 
the basis on which teachers volunteered to participate or the basis on which school senior leaders 
selected participants. The third question (10c) asked teachers whether they agreed with the question 
‘| feel under pressure if colleagues discuss my teaching’. The likelihood that teachers participating in 
the project would disagree with this statement increased between the baseline and end of project 
surveys (p<.01). Interestingly, teachers who were not participating in the project were also more likely 
to disagree with the statement ‘I feel under pressure if colleagues discuss my teaching’ at the end of 
the project. There is some tentative evidence here that the intervention had affected the climate for 
collaboration in the schools. Nonetheless, even at the end of the project, a third of participants and 
four tenths of non-participants still agreed that they felt under pressure if colleagues discussed their 
teaching. 


Table A5F.3 — Teachers’ views about working with and learning from colleagues 


ae Disagree | Unsure | Agree ee 
10a | can trust : Project 0.0% 1.6% 4.8% 53.2% 40.3% 
my colleagues | Baseline Non-Project | 0.0% 0.0% 3.1% | 53.1% | 43.8% 
to be supportive - ; - F = - 
if they observe Endpoint Project 0.0% 1.6% 1.6% 54.0% | 42.9% 
my teaching Non-Project | 0.0% 6.7% 3.3% | 50.0% | 40.0% 
ae Project 0.0% 0.0% 6.5% 38.7% | 54.8% 
aseline 
ae learn a lot Non-Project | 0.0% 0.0% 3.1% | 62.5% | 34.4% 
rom m 
salesoies Eadacini Project 0.0% 0.0% 4.8% 46.0% | 49.2% 
Non-Project 0.0% 3.3% 6.7% 50.0% | 40.0% 
10c | feel under hed Project 3.2% 24.2% 274% | 37.1% 8.1% 
aseline 
pressure if Non-Project | 9.4% 25.0% 21.9% | 43.8% | 0.0% 
colleagues 
discuss my Ena Project 3.2% 46.0% 17.5% | 30.2% 3.2% 
teaching i Non-Project 3.3% 43.3% 13.3% | 36.7% 3.3% 
10d | find it Baseline Project 1.6% 47.5% 23.0% | 19.7% 8.2% 
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sy useful to Non-Project 0.0% 40.6% 25.0% | 21.9% | 12.5% 

reflect on my : 

lessons on my | Endpoint Project 6.3% 38.1% 23.8% | 254% | 6.3% 
own Non-Project | 3.3% 30.0% 30.0% | 30.0% | 6.7% 


Total n=88-95, Of which participants n=59-63, non-participants n=29-32 


Use of IRIS-CONNECT by project participants and by other teachers in the school 


Part of the end of project survey was only answered by 89 teachers who had made some use of IRIS- 
CONNECT. 69 of these teachers reported that they had participated in the project and 20 reported 
they had not participated in the project. Table A5F.4 summarises these teachers’ responses to 
questions asking them how they had used IRIS-CONNECT. Teachers who had not participated in the 
project were more likely to report that they had used IRIS-CONNECT in the context of a group of 
teachers as opposed to working (e.g. in coaching or mentoring) with one other teacher. Teachers who 
had participated in the project reported high levels of use, particularly in a group format as 
encouraged by the intervention. 


Table A5F.4 — How teachers used IRIS-CONNECT 


Never Once or twice | 3-4 occasions 201 nee 
occasions 

11a In-ear coaching (being Non-project 16 3 1 0 
coached) Project 59 9 0 1 
11b In-ear coaching (as a Non-project 19 0 0 0 
coach) Project 61 4 1 ik 
11c 1-1 mentoring (being Non-project 13 2 4 1 
mentored) Project 48 13 4 1 
11d 1-1 mentoring (as a Non-project 17 1 1 0 
mentor) Project 45 15 3 1 
11e In a group viewing and Non-project 13 6 1 0 
discussing an example of 
practice from another school Project 6 23 26 13 
11f In a group viewing and Non-project 9 5 6 0 
discussing an example from 
a member of the group Project 2 26 33 7 
11g In a group sharing in 
fecemine alae and Non piolee 3 : 0 
ee a en Project 26 20 17 5 


Teachers were also asked about the contexts in which they had used IRIS-CONNECT (see Table 
A5F.5). These responses show that just over a third of project participants had used IRIS-CONNECT 
in the contexts of supporting trainee teachers and newly qualified teachers (NQTs). A substantial 
minority of teachers participating in the project had experience of using video technology in other 
contexts. This experience might be expected to foster their willingness to use the technology and their 
knowledge of how to use the technology effectively to aid their learning. Participants were more likely 
than non-participants to report that they had used IRIS-CONNECT to support the development of new 
teachers. Conversely, non-participants were more likely than participants to report that they had used 
IRIS-CONNECT in the context of a school’s appraisal system. The numbers are too small here to 
draw any firm inferences, but there is an indication here that there may be a tension between use of 
the technology for two approaches to school improvement: evaluation of performance or development 
of thinking and practice. The level of participants’ reported use of IRIS-CONNECT in collaborative 
CPD suggests that most of these teachers regarded their participation as relevant to their job. 
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Table A5F.5 The contexts for schools’ use of IRIS-CONNECT 


IRIS Connect 


Participated | Participated | Participated 
Not but not and modest as major 
participated | important to part of my part of my 
my job job job 
12a Supporting trainee Non-project 17 0 2 1 
teachers Project 45 6 9 6 
Non-project 17 1 2 0 
12b Support for NQTs Project 43 6 9 10 
12c As part of the Non-project 7 2 4 5 
school’s appraisal system Project 36 3 20 7 
12d As part of the - Non-project 4 2 8 5 
school’s collaborative : 
CPD programme Project 8 6 42 13 
12e As part of a Non-project 16 0 2 0 
collaboration between 
Sonia mieut Project 48 3 11 4 
partnership 


Teachers’ beliefs about the feasibility and value of the intervention 


Teachers were generally positive about the technical and practical feasibility of using IRIS-CONNECT 
and a large majority agreed that it was easy to use the online platform to review lessons. The end of 
project survey asked teachers to reflect on their experience of IRIS-CONNECT. A minority of teachers 
(about one in seven) thought that preparation for film clubs had taken too much of their time. We 
received between 84 and 90 responses to these questions. Close to three-quarters of participating 
teachers regarded the project as good use of their time and effective CPD (see questions 4 and 9). 
Only a very small minority believed that their time had not been well used. More than 80% believed 
that the focus on formative assessment aligned well with their professional development needs. About 
70% believed that the film clubs had made them more likely to reflect on their own practice, but 
teachers were less sure about the impact of the intervention on their own future use of IRIS- 
CONNECT and future collaboration (though more than half of teachers had positive expectations). 


Table A5F.6 Teachers’ beliefs about value and practicality of the intervention 


St I St I Total 
Statement ee Y Disagree | Unsure | Agree dela ies 
Disagree Agree Responses 

1 It is easy to use IRIS-Connect N 2 15 13 52 8 90 

cameras to record videos % 2.2% 16.7% 14.4% | 57.8% | 8.9% 

It is easy to use the IRIS-Connect N 1 7 4 63 15 90 
2 | platform to view my recorded 

vidiews % 1.1% 7.8% 4.4% | 70.0% | 16.7% 

The amount of preparation needed | N 3 44 26 12 1 86 
3 | for film clubs has been too much to 

manage % 3.5% 51.2% 30.2% | 14.0% 1.2% 
4 The time | have spent using IRIS- N 0 5 20 50 14 89 

Connect has been worth it % 0.0% 5.6% 22.5% | 56.2% | 15.7% 

Using IRIS-Connect has become an N (@) 20 20 43 7 90 
5 | important part of my professional 

development practice % 0.0% 22.2% 22.2% | 47.8% 7.8% 

The focus on formative feedback N 0 5 11 61 11 88 
6 | has fit within my own professional 

development needs % 0.0% 5.7% 12.5% | 69.3% | 12.5% 

Attending film clubs has made me N 0 7 18 50 10 85 
7 | more likely to reflect on my own 

practice % 0.0% 8.2% 21.2% | 58.8% | 11.8% 
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3 Film clubs have led to increased N 0 9 25 36 14 84 
teacher collaboration in my school % 0.0% 10.7% 29.8% | 42.9% 16.7% 

9 Film clubs have been an effective N 0 2 21 50 12 85 
approach to professional learning % 0.0% 2.4% 24.7% | 58.8% 14.1% 


Open responses in the end of project survey 


This section summarises the responses of 5 open-ended items from the teacher end of project survey 


Following the feedback characterisation scale, respondents were asked about the main factors 
affecting their choices to give different feedback in different circumstances. 


Many teachers altered their typical feedback according to the following main factors: 

o Stage within a curriculum: several teachers vary their feedback according to the stage that has 
been reached within a sequence of lessons. For several teachers this was over the course of the 
year, with greater weight being put on scaffolding and focused feedback in earlier terms, with a 
greater emphasis on challenge, extension and making links in later terms. 

0 Type of feedback: many respondents noted that their written and verbal feedback tends to differ, 
with verbal feedback being a better way of questioning and getting the children to review their 
steps. Written feedback is, according to one respondent, ‘generally more direct’. 

o Differences between children: The most common group of responses to this item related to 
differences between children. First, many respondents noted that feedback related to level of 
understanding or ability. Comments related to the idea that learning goes from basic to complex 
and feedback should reflect this (also see first point, above). The other major area where 
respondents thought different feedback was appropriate related to the needs of pupils and pupil 
groups. In particular, pupils with English as an additional language and younger pupils require the 
teacher to ensure that feedback is appropriate. 


Has IRIS-Connect helped you make changes to your classroom practice? 
(Please give details/examples) 


The feedback from this item was overwhelmingly positive. Of 84 comments there were only 4 not 
unambiguously positive. Of these 4, 2 were simply ‘no’, 1 respondent said that he or she had not used 
it enough to be sure and 1 thought there was an impact in the short run but not in the long term. The 
80 positive comments can be grouped into several, sometimes overlapping, areas (in approximate 
order of how frequently these were mentioned): 

0 Questioning: the most frequently referred to area of improvement was questioning. Improvements 
respondents commented on include a greater awareness of the questions they would ask before 
teaching and would plan effective questions, that their questions tended to be more open-ended, 
that they had made changes in the precise language they use to encourage dialogue and that 
they were more able to focus questions to the need of a particular group. 

o Feedback and teacher language: there were numerous general and miscellaneous comments 
about feedback and language. These included comments on the tone of voice, facial expressions 
and non-verbal feedback, feedback being more constructive, building greater opportunities for 
feedback into tasks and the elimination of bad habits (e.g. saying ‘okay’ repeatedly). 

0 Teacher CPD, reflection and awareness: a large number of respondents thought IRIS has made 
them more reflective, more aware of various aspects of their practice and of the needs of their 
pupils. There are many comments drawing attention to the ownership teachers get of their 
professional development, IRIS is referred to as ‘allowing’ or ‘enabling’ them to review their own 
lessons, identify their own areas for improvement, to implementing things learnt from other 
teachers and to monitor improvements in their own teaching. 

0 Personalising learning and identifying needs: many respondents commented on using IRIS for 
planned observation of individual children or groups of children (e.g. pupil premium or vulnerable 
children) and gaining understanding of the varying needs of individual children in their class. 
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o Pupil talk: respondents commented on using their questioning to direct children to talk to each 
other rather than themselves, giving pupils longer to answer (wait time), get better explanations 
from pupils and increases in the amount of peer talk and collaboration in their classrooms. 

0 Behaviour management and routines: there were several comments about improvements in 
classroom routines, general lesson delivery and behaviour management. 


Have you changed how you think about feedback as a result of using IRIS-Connect? (Please give 
details/examples) 


Many of the points overlap with those discussed in the previous question. There are a couple of points 

which come out more strongly, or in more detail in this question specifically on feedback: 

0 Open ended discussions: a large number of responses related to one of the key ideas in the first 
module that questions can be used to open up dialogue and often feedback can close it down. 
Respondents commented that they now ‘hold [their] opinion back’ and one opened their comment 
with, ‘Don’t always give the answers!’. In the same vein, several teachers noted that they are 
trying not to give non-verbal clues or be too leading in their feedback to always sustain rather than 
close discussion. 

0 Better and/or more child talk: a large number of comments related to changes to the amount or 
quality of child talk in the respondents’ classrooms. Teachers commented on thinking about their 
feedback as a way of encouraging explanation and justification from the children. Also, many 
commented on pupils having greater opportunity to respond to each other, pupil ownership of the 
discussion and feedback encouraging pupils to identify and work through their misconceptions to 
a greater extent. 


Have you had a particular focus/purpose with regards to your use of IRIS Connect? 


This item ties in with the previous two questions, supporting the idea that feedback, questioning, pupil 
talk and classroom dialogue in general are the main foci for teachers. Also, as in previous questions 
there are some mentions of behaviour management, pace and delivery. 


Would you continue to use IRIS-Connect in the future if the option was available to you? If yes - 
would you use it in the same way as now? If no - why not? 


The vast majority of responses to this question were positive, with about half a dozen respondents not 

sure or not wanting to continue use. There were a sizable minority who wanted to continue to use it 

but followed this with a caveat or reservation. The responses can be roughly grouped as follows, 
sorted approximately into descending order according to the number of respondents commenting in 
each area: 

o Yes, the same or more of the same: Many left a positive comment and wanted their use of IRIS- 
Connect to continue as now. Some commented that they wanted the same but with a difference 
focus, a wider range of lessons or an increased number or string of lessons. A large number 
simply responded ‘yes’ with no additional informative comment. 

0 Yes, with a particular focus: several teachers wanted to continue to use IRIS for a given purpose 
including, for literacy, to help particular children or groups or for tying new strategies out. 

0 Yes, but with a caveat, reservation or alteration: several teachers broadly agreed before giving a 
reservation, or proviso such as needing more collaboration, wanting to work in year groups or 
pairs, only after resolving technical issues and — as mentioned several times — if sufficient time is 
allocated for use and for film clubs. 

0 Yes, for NQTs/coaching mentoring: There were several mentions of continued use of IRIS- 
Connect for coaching and mentoring purposes, in many cases specifically for NQTs. 
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Appendix 5G: Project leader end of project survey 


At the end of the project a survey (see Appendix 3C) was sent to school champions (separately from 
the main teachers’ survey reported in Appendix 5F). Although there were 11 schools we received 
three returns of the school champion survey from one school, giving a total of 13. The survey 
contained some Likert scale items (summarised in Table A5G.1) and a number of open ended items. 
School champions thought that using IRIS-Connect had increased the amount of collaborative 
professional learning at their school, despite not all staff being enthusiastic about using it at first. 
School champions believed that enough guidance and information was given for them to effectively 
run the film clubs. They also believed that the focus on feedback was suitable for the needs of their 
school and that staff positively engaged with the film clubs. On the whole, school champions agreed 
that the content on the IRIS-CONNECT platform was stimulating and that film clubs were effective for 
whole-school CPD. However, one respondent, disagreeing with both of these statements and 2 
disagreed with the second (both from the same school). 


Table A5G.1 School champions reporting of the operation of the intervention 


Strongly Strongly 


Disagree Disagree | Unsure | Agree Agree Missing 
Observing others is an important part of my 
la | school's approach to professional 0 0 1 4 8 0 
development 
1b Staff at my school were initially enthusiastic 0 3 A 5 1 0 
about the prospect of using IRIS-Connect 
Film clubs are an effective way of running 
ie whole-school CPD ‘ : 2 g 
4d The focus on feedback was suitable for my 0 0 1 j 5 0 


school’s needs 


Using IRIS-Connect has increased the 
le | amount of collaborative professional 0 0 0 8 5 0 
learning taking place at my school 


| was given enough guidance and 


1f | information to be able to effectively run film 0 0 0 6 7 0 
clubs 
The online content on the IRIS-Connect 

1g | platform to support film clubs was 0 2 0 4 7 0 
stimulating 


Teachers at my school positively engaged 


1h | with the film clubs : a : 2 ° : 

da | had to spend a lot of time to organise and 1 6 0 6 0 0 
set-up film clubs 

4b After the first film club, organising 0 3 3 3 0 0 
subsequent film clubs was much quicker 

ne Using IRIS-Connect has been good value- 0 1 > 3 1 1 
for money 

Ad Film clubs are a sustainable approach to 0 0 1 3 3 1 
whole-school professional development 

de The success of film clubs is highly 0 3 A 6 1 0 


dependent on the project leader’s input 


There was an even split between school champions who thought that it took a lot of time to organise 
and set up film clubs, although most thought that this was quicker after the first film club. Most school 
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champions thought that using IRIS-CONNECT is good value for money, with 2 respondents unsure 
and 1 disagreeing (who also disagreed about the effectiveness of the approach, see above). 

All respondents apart from 1 thought that film clubs were a sustainable approach to whole-school 
professional development. The one respondent who was unsure referred to the time constraints and 
trying to work around hectic school schedules in order to be able to get a suitable time for everyone 
to attend film clubs.’ This respondent recommended that fewer film club meetings should be required. 
Finally, there was some agreement that the success of film clubs was dependent on the input of the 
project leader but many were unsure and 2 respondents disagreed. 


Reponses from Open-Ended Questions: 


What aspect(s) of your film clubs do you think was particularly effective? 


e The main points picked out related to a) watching videos of other teachers, b) discussion of clips 
in general and c) discussion of teachers’ own clips. There were some mentions of the platform 
content such as the theory/pre-reading on the platform or the ability to comment but these 
seemed a smaller but still appreciable part of the overall picture. 

e The value of clips from various sources is mentioned: unknown teachers (exemplars of theory into 
practice, non-threatening), of self (having colleagues discuss own teaching, self-reflection) and 
other teachers in school (learn from colleagues, share good practice). 

e There was a general sense about the value of professional dialogue and the project bringing this 
about in relation to clips in general and clips of their own lessons. As might be expected given the 
intervention, the idea of watching and discussing clips as a group cuts across most comments. 

e There are also a couple of comments about the ‘chance’/opportunity’/being ‘allowed’ to 
reflect/discuss practice 


How would you change film clubs for any future use? 


e The main points raised related to time commitment (5 respondents) and having a greater 
emphasis on their own videos (as opposed to those provided by IRIS) (3 respondents). 

e Two respondents did not want any change at all. 

e One suggested an improvement to the content (more year groups and subject coverage). 

e One made acomment about the design (change name and use small teams). As one of the 
respondents who did not think changes were needed pointed out, however, the process is quite 
flexible. 

e = In relating to timings/amount, fitting in the film clubs was a clear difficulty (see next question also). 
There was also a point made about the amount of material to work through. 

e The need for more/earlier emphasis came up for 3 respondents. This was also something 
mentioned in other data (e.g. telephone interviews and reflected in the metrics data). 


What have been the most time-consuming aspects of organising film clubs? 


e Relating to the point in the previous question about fitting in film clubs, a key point raised by most 
respondents relates to finding appropriate times for staff to meet. 

e Acouple of respondents mentioned the time to organise for the equipment to be available when 
needed. 

e There was only one mention of the project leader’s preparation time for film clubs (presumably as 
the content on the platform was already ready-made). 

e There was one mention of the time taken to find people to agree to share clips at film club. 


Would you like to continue to use IRIS-Connect in the future (or already have plans to do so)? If yes - 
would you use it in the same way as now? If no, please explain why not. 


e All but one respondent wanted to continue to use IRIS-Connect. This one respondent was one of 
three respondents from a single school where one other said they would wish to continue using it. 

e Most respondents wanted to carry on running film clubs, with many intended to extend it to more 
staff where all staff were not already involved. 

e One respondent said they would like to embed IRIS into their appraisal process. 
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e One respondent listed several uses: self-reflection, NQT development, sharing practice. 


What are the problems with using videos for professional development? 


e Many raised concerns about the reluctance of some staff to be filmed. One thought that this 
improved over time but one school had not resolved the problem. Similarly, one respondent 
pointed out that parent may not want videos shared outside of the school. 

e One respondent pointed out that editing clips to find the most meaningful parts can be time 
consuming. 

e Several respondents commented about technical problems relating to a) sound quality, b) internet 
connectivity and c) targeting the video. Three of the 7 comments were from the single school with 
three respondents. 


Have you noticed any clear changes in the practice or attitudes of teachers in your school? 


e Most respondents commented about the improvements in the collaborative culture in the school. 
Many comments relate to teachers being enthusiastic, discussing their practice and wanting to 
use IRIS to improve their practice. 

e There are also comments about specific changes in practice relating to the project focus of 
feedback and dialogue. 
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Appendix 5H: Teacher Interviews 


This appendix summarises the main points raised by teachers (who were not school champions) 


interviewed at the end of the project: 


Overall design and components of the project: There was a broad consensus that the various 
aspects of the intervention worked well together. Several respondents articulated how watching 
the videos led to discussion that were framed by the content and questions and how this had led 
to reflection, changes in thinking and improvements in practice. The discussion of videos (rather 
than merely watching the videos) was seen as essential as this encourages engagement, creates 
greater awareness of the important aspects of the practice being observed and lets differences of 
opinion in the group arise to create discussion (e.g. stemming from different teaching styles or 
philosophies et cetera). 

More or less effective aspects: While the aspects of the project were thought to work well together 
(see above), several respondents picked out more or less useful aspects. There is a suggestion 
that respondents found watching the clips and the resulting group discussions more useful than 
the written content on the classroom. The written content, however, was still commented on as 
being useful, especially in its role in helping frame the discussion. There was a mix of views about 
the relative benefit of videos of their own teaching compared to that of others. 

Most effective aspects: The most frequently mentioned single aspect of the project mentioned 
was a video in the first module of an American teacher doing a true/false mathematics activity. 
This was picked out by many respondents. This video was described by one teacher as the 
‘foundation building block’ for the entire project, as it so clearly exemplified the dialogic approach 
to be developed. There are many mentions of various aspects of this clip, from the general 
approach, the task and the language used and one respondent attempted an almost exact 
replica, with a slightly different mathematics task (this is shared with us). Other learning points 
that were frequently raised included the ‘pose, pause, pounce, bounce’ approach to questioning, 
use of greater wait time, aspects of how questions were phrased and the general approach of not 
restricting the discussion with feedback the closes it down to the correct answer 

Collaborative CPD Culture: there were several mentions of the culture required for sharing videos 
and the reluctance many felt about others observing their own practice. This was sometimes 
explicitly mentioned to say that others did not wish to share videos. Sometimes this was 
mentioned as being initial reservations that were settled as the project went on. One respondent 
discussed how there was a lot of initial enthusiasm with ‘a sort of buzz’ around the school as 
teachers were ‘desperate’ to try things out for themselves. The school project leader, however, 
was also credited with managing this and encouraging teachers to identify a focus and share 
videos in smaller groups at a later stage. 

Complementary and competing CPD: Respondents at one school were doing the project 
alongside a pre-existing lesson study approach to individual professional development. This 
lesson study fit into the EEF feedback project and so this was seen to be complimentary, for one 
in particular, the focus on feedback was ideal as this fit within their pre-existing professional 
development focus. One respondent had run a school INSET on feedback which also 
complemented the project. 

Impact: respondents were generally very positive about the impact of the project and many 
specific and general changes in practice were mentioned. These changes are all in line with those 
raised in the teacher surveys (see above) mainly relating to feedback, dialogue, pupil ownership 
of discussion and questioning. For some, only minor changes were made due to their feedback 
practice already having many of the features promoted in the programme; even in this case, 
however, specific improvement points were identified. One respondent had made changes to their 
practice in relation to group work but apparently had not taken the approach more widely. Other 
teachers credited the project with significant and widespread changes in their practice, with the 
approach or aspects of it being evident ‘daily’ and across many aspects of their teaching. There is 
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some mention of a need to shift the classroom culture and get the pupils used to responding to 
each other and in greater depth. In terms of the speed of changes in classroom practice, one 
respondent commented that it was a gradual process but not a ‘huge amount of time’. 

e Negatives: 


Oo 


The most common and significant criticism to arise relates to time. One respondent who 
had been deliberately contacted due to a survey suggesting good engagement with the 
project, yet a fairly negative overall judgement explained that the project had taken too 
much time for little benefit to them. They were not given any further time to do the project 
and were expected to reflect on their videos and complete film clubs outside of school 
time alongside all other activities. Time was also raised by other respondents, in a much 
more positive light, however. One respondent would have liked to have been given more 
time and thought that the time demands were not a problem with the project per se, but 
just related to the already heavy demands placed on schools. Getting film clubs 
organised was mentioned as a particular point of difficulty. Some respondents had had 
time set aside during staff meeting time or through release time specifically for the 
project, they credited this as being important to the amount of success they have had. 
The other negative item raised by the respondent purposively sampled to understand a 
more negative perspective on the project thought that the standard of some of the clips 
was low and that teaching of such a low standard would not have been accepted at their 
school. It is interesting to contrast this to other respondents who had mentioned the 
variable standard of clips in a positive light: that they found it useful to discuss a more 
typical example of the approach and identify where improvements could be made. 
Another explicitly noted the value in ‘picking apart’ some of the clips to identify strengths 
and weaknesses. There seems to have been a difference in the assumptions about what 
the clips were supposed to show, with the negative respondent expecting all clips to be 
exemplary. 


e Improvements: The respondents were asked what if anything they would change about the 
project. Note that this is a very small sample and the teacher/project leader survey gives a more 
comprehensive picture. The following suggestions arose: 


Oo 


Having a larger bank of shorter clips, especially those of a high standard/from own 
colleagues. 

Increasing the relevance of the project to pupils of different ages. One respondent 
especially thought that some of the clips from Key Stage 2 were less relevant for their 
Key Stage 1 group, and would have liked more of the latter. Another respondent, 
however, explicitly commented that they liked the range or age groups. 

Having less theory and written content. The written content was generally thought to be 
useful, but one respondent thought that the balance needed to be slightly shifted towards 
videos and supporting questions. 

Some respondents commented that they would have liked to see more videos of other 
staff and one in particular firmly believed that the project needed a greater emphasis, 
maybe with a greater level of compulsion, for videos to upload their own videos to 
demonstrate improvement in a given area of practice and more widely for discussion 
purposes. 
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